Namespace faiss¶
-
namespace faiss
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree.
Throughout the library, vectors are provided as float * pointers. Most algorithms can be optimized when several vectors are processed (added/searched) together in a batch. In this case, they are passed in as a matrix. When n vectors of size d are provided as float * x, component j of vector i is
x[ i * d + j ]
where 0 <= i < n and 0 <= j < d. In other words, matrices are always compact. When specifying the size of the matrix, we call it an n*d matrix, which implies a row-major storage.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. I/O functions can read/write to a filename, a file handle or to an object that abstracts the medium.
The read functions return objects that should be deallocated with delete. All references within these objectes are owned by the object.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. Definition of inverted lists + a few common classes that implement the interface.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. Since IVF (inverted file) indexes are of so much use for large-scale use cases, we group a few functions related to them in this small library. Most functions work both on IndexIVFs and IndexIVFs embedded within an IndexPreTransform.
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. In this file are the implementations of extra metrics beyond L2 and inner product
Copyright (c) Facebook, Inc. and its affiliates.
This source code is licensed under the MIT license found in the LICENSE file in the root directory of this source tree. Defines a few objects that apply transformations to a set of vectors Often these are pre-processing steps.
Typedefs
-
using IndexReplicas = IndexReplicasTemplate<Index>
-
using IndexBinaryReplicas = IndexReplicasTemplate<IndexBinary>
-
using IndexShards = IndexShardsTemplate<Index>
-
using IndexBinaryShards = IndexShardsTemplate<IndexBinary>
-
using ConcatenatedInvertedLists = HStackInvertedLists
-
using IndexIDMap = IndexIDMapTemplate<Index>
-
using IndexBinaryIDMap = IndexIDMapTemplate<IndexBinary>
-
using IndexIDMap2 = IndexIDMap2Template<Index>
-
using IndexBinaryIDMap2 = IndexIDMap2Template<IndexBinary>
Enums
-
enum MetricType
The metric space for vector comparison for Faiss indices and algorithms.
Most algorithms support both inner product and L2, with the flat (brute-force) indices supporting additional metric types for vector comparison.
Values:
-
enumerator METRIC_INNER_PRODUCT
maximum inner product search
-
enumerator METRIC_L2
squared L2 search
-
enumerator METRIC_L1
L1 (aka cityblock)
-
enumerator METRIC_Linf
infinity distance
-
enumerator METRIC_Lp
L_p distance, p is given by a faiss::Index metric_arg
-
enumerator METRIC_Canberra
some additional metrics defined in scipy.spatial.distance
-
enumerator METRIC_BrayCurtis
-
enumerator METRIC_JensenShannon
-
enumerator METRIC_INNER_PRODUCT
Functions
-
Quantizer *clone_Quantizer(const Quantizer *quant)
-
float kmeans_clustering(size_t d, size_t n, size_t k, const float *x, float *centroids)
simplified interface
- Parameters:
d – dimension of the data
n – nb of training vectors
k – nb of output centroids
x – training set (size n * d)
centroids – output centroids (size k * d)
- Returns:
final quantization error
-
Index *index_factory(int d, const char *description, MetricType metric = METRIC_L2)
Build and index with the sequence of processing steps described in the string.
-
IndexBinary *index_binary_factory(int d, const char *description)
-
void write_index(const Index *idx, const char *fname)
-
void write_index(const Index *idx, FILE *f)
-
void write_index(const Index *idx, IOWriter *writer)
-
void write_index_binary(const IndexBinary *idx, const char *fname)
-
void write_index_binary(const IndexBinary *idx, FILE *f)
-
void write_index_binary(const IndexBinary *idx, IOWriter *writer)
-
Index *read_index(const char *fname, int io_flags = 0)
-
Index *read_index(FILE *f, int io_flags = 0)
-
Index *read_index(IOReader *reader, int io_flags = 0)
-
IndexBinary *read_index_binary(const char *fname, int io_flags = 0)
-
IndexBinary *read_index_binary(FILE *f, int io_flags = 0)
-
IndexBinary *read_index_binary(IOReader *reader, int io_flags = 0)
-
void write_VectorTransform(const VectorTransform *vt, const char *fname)
-
VectorTransform *read_VectorTransform(const char *fname)
-
ProductQuantizer *read_ProductQuantizer(const char *fname)
-
ProductQuantizer *read_ProductQuantizer(IOReader *reader)
-
void write_ProductQuantizer(const ProductQuantizer *pq, const char *fname)
-
void write_ProductQuantizer(const ProductQuantizer *pq, IOWriter *f)
-
void write_InvertedLists(const InvertedLists *ils, IOWriter *f)
-
InvertedLists *read_InvertedLists(IOReader *reader, int io_flags = 0)
-
void initialize_IVFPQ_precomputed_table(int &use_precomputed_table, const Index *quantizer, const ProductQuantizer &pq, AlignedTable<float> &precomputed_table, bool verbose)
Pre-compute distance tables for IVFPQ with by-residual and METRIC_L2
- Parameters:
use_precomputed_table – (I/O) =-1: force disable =0: decide heuristically (default: use tables only if they are < precomputed_tables_max_bytes), set use_precomputed_table on output =1: tables that work for all quantizers (size 256 * nlist * M) =2: specific version for MultiIndexQuantizer (much more compact)
precomputed_table – precomputed table to initialize
-
inline uint64_t lo_build(uint64_t list_id, uint64_t offset)
-
inline uint64_t lo_listno(uint64_t lo)
-
inline uint64_t lo_offset(uint64_t lo)
-
template<int A = 32>
inline bool is_aligned_pointer(const void *x)
-
float fvec_L2sqr(const float *x, const float *y, size_t d)
Squared L2 distance between two vectors.
-
float fvec_inner_product(const float *x, const float *y, size_t d)
inner product
-
float fvec_L1(const float *x, const float *y, size_t d)
L1 distance.
-
float fvec_Linf(const float *x, const float *y, size_t d)
infinity distance
-
void pairwise_L2sqr(int64_t d, int64_t nq, const float *xq, int64_t nb, const float *xb, float *dis, int64_t ldq = -1, int64_t ldb = -1, int64_t ldd = -1)
Compute pairwise distances between sets of vectors
- Parameters:
d – dimension of the vectors
nq – nb of query vectors
nb – nb of database vectors
xq – query vectors (size nq * d)
xb – database vectors (size nb * d)
dis – output distances (size nq * nb)
ldq, ldb, ldd – strides for the matrices
-
void fvec_inner_products_ny(float *ip, const float *x, const float *y, size_t d, size_t ny)
-
void fvec_L2sqr_ny(float *dis, const float *x, const float *y, size_t d, size_t ny)
-
size_t fvec_L2sqr_ny_nearest(float *distances_tmp_buffer, const float *x, const float *y, size_t d, size_t ny)
-
float fvec_norm_L2sqr(const float *x, size_t d)
squared norm of a vector
-
void fvec_norms_L2(float *norms, const float *x, size_t d, size_t nx)
compute the L2 norms for a set of vectors
- Parameters:
norms – output norms, size nx
x – set of vectors, size nx * d
-
void fvec_norms_L2sqr(float *norms, const float *x, size_t d, size_t nx)
same as fvec_norms_L2, but computes squared norms
-
void fvec_renorm_L2(size_t d, size_t nx, float *x)
-
void inner_product_to_L2sqr(float *dis, const float *nr1, const float *nr2, size_t n1, size_t n2)
-
void fvec_add(size_t d, const float *a, const float *b, float *c)
compute c := a + b for vectors
c and a can overlap, c and b can overlap
- Parameters:
a – size d
b – size d
c – size d
-
void fvec_add(size_t d, const float *a, float b, float *c)
compute c := a + b for a, c vectors and b a scalar
c and a can overlap
- Parameters:
a – size d
c – size d
-
void fvec_sub(size_t d, const float *a, const float *b, float *c)
compute c := a - b for vectors
c and a can overlap, c and b can overlap
- Parameters:
a – size d
b – size d
c – size d
-
void fvec_inner_products_by_idx(float *ip, const float *x, const float *y, const int64_t *ids, size_t d, size_t nx, size_t ny)
-
void fvec_L2sqr_by_idx(float *dis, const float *x, const float *y, const int64_t *ids, size_t d, size_t nx, size_t ny)
-
void pairwise_indexed_L2sqr(size_t d, size_t n, const float *x, const int64_t *ix, const float *y, const int64_t *iy, float *dis)
compute dis[j] = L2sqr(x[ix[j]], y[iy[j]]) forall j=0..n-1
- Parameters:
x – size (max(ix) + 1, d)
y – size (max(iy) + 1, d)
ix – size n
iy – size n
dis – size n
-
void pairwise_indexed_inner_product(size_t d, size_t n, const float *x, const int64_t *ix, const float *y, const int64_t *iy, float *dis)
-
void knn_inner_product(const float *x, const float *y, size_t d, size_t nx, size_t ny, float_minheap_array_t *res)
Return the k nearest neighors of each of the nx vectors x among the ny vector y, w.r.t to max inner product.
- Parameters:
x – query vectors, size nx * d
y – database vectors, size ny * d
res – result heap structure, which also provides k. Sorted on output
-
void knn_inner_product(const float *x, const float *y, size_t d, size_t nx, size_t ny, size_t k, float *distances, int64_t *indexes)
Return the k nearest neighors of each of the nx vectors x among the ny vector y, for the inner product metric.
- Parameters:
x – query vectors, size nx * d
y – database vectors, size ny * d
distances – output distances, size nq * k
indexes – output vector ids, size nq * k
-
void knn_L2sqr(const float *x, const float *y, size_t d, size_t nx, size_t ny, float_maxheap_array_t *res, const float *y_norm2 = nullptr)
Return the k nearest neighors of each of the nx vectors x among the ny vector y, for the L2 distance
- Parameters:
x – query vectors, size nx * d
y – database vectors, size ny * d
res – result heap strcture, which also provides k. Sorted on output
y_norm2 – (optional) norms for the y vectors (nullptr or size ny)
-
void knn_L2sqr(const float *x, const float *y, size_t d, size_t nx, size_t ny, size_t k, float *distances, int64_t *indexes, const float *y_norm2 = nullptr)
Return the k nearest neighors of each of the nx vectors x among the ny vector y, for the L2 distance
- Parameters:
x – query vectors, size nx * d
y – database vectors, size ny * d
distances – output distances, size nq * k
indexes – output vector ids, size nq * k
y_norm2 – (optional) norms for the y vectors (nullptr or size ny)
-
void knn_inner_products_by_idx(const float *x, const float *y, const int64_t *ids, size_t d, size_t nx, size_t ny, float_minheap_array_t *res)
-
void knn_L2sqr_by_idx(const float *x, const float *y, const int64_t *ids, size_t d, size_t nx, size_t ny, float_maxheap_array_t *res)
-
void range_search_L2sqr(const float *x, const float *y, size_t d, size_t nx, size_t ny, float radius, RangeSearchResult *result)
Return the k nearest neighors of each of the nx vectors x among the ny vector y, w.r.t to max inner product
- Parameters:
x – query vectors, size nx * d
y – database vectors, size ny * d
radius – search radius around the x vectors
result – result structure
-
void range_search_inner_product(const float *x, const float *y, size_t d, size_t nx, size_t ny, float radius, RangeSearchResult *result)
same as range_search_L2sqr for the inner product similarity
-
void compute_PQ_dis_tables_dsub2(size_t d, size_t ksub, const float *centroids, size_t nx, const float *x, bool is_inner_product, float *dis_tables)
specialized function for PQ2
-
void pairwise_extra_distances(int64_t d, int64_t nq, const float *xq, int64_t nb, const float *xb, MetricType mt, float metric_arg, float *dis, int64_t ldq = -1, int64_t ldb = -1, int64_t ldd = -1)
-
void knn_extra_metrics(const float *x, const float *y, size_t d, size_t nx, size_t ny, MetricType mt, float metric_arg, float_maxheap_array_t *res)
-
FlatCodesDistanceComputer *get_extra_distance_computer(size_t d, MetricType mt, float metric_arg, size_t nb, const float *xb)
get a DistanceComputer that refers to this type of distance and indexes a flat array of size nb
- SPECIALIZED_HC (4)
- SPECIALIZED_HC (8)
- SPECIALIZED_HC (16)
- SPECIALIZED_HC (20)
- SPECIALIZED_HC (32)
- SPECIALIZED_HC (64)
-
inline int generalized_hamming_64(uint64_t a)
-
void generalized_hammings_knn_hc(int_maxheap_array_t *ha, const uint8_t *a, const uint8_t *b, size_t nb, size_t code_size, int ordered = true)
generalized Hamming distances (= count number of code bytes that are the same)
-
void bitvec_print(const uint8_t *b, size_t d)
-
void fvecs2bitvecs(const float *x, uint8_t *b, size_t d, size_t n)
-
void bitvecs2fvecs(const uint8_t *b, float *x, size_t d, size_t n)
-
void fvec2bitvec(const float *x, uint8_t *b, size_t d)
-
void bitvec_shuffle(size_t n, size_t da, size_t db, const int *order, const uint8_t *a, uint8_t *b)
Shuffle the bits from b(i, j) := a(i, order[j])
-
inline int popcount64(uint64_t x)
-
void hammings(const uint8_t *a, const uint8_t *b, size_t na, size_t nb, size_t nbytespercode, hamdis_t *dis)
Compute a set of Hamming distances between na and nb binary vectors
- Parameters:
a – size na * nbytespercode
b – size nb * nbytespercode
nbytespercode – should be multiple of 8
dis – output distances, size na * nb
-
void hammings_knn_hc(int_maxheap_array_t *ha, const uint8_t *a, const uint8_t *b, size_t nb, size_t ncodes, int ordered)
Return the k smallest Hamming distances for a set of binary query vectors, using a max heap.
- Parameters:
a – queries, size ha->nh * ncodes
b – database, size nb * ncodes
nb – number of database vectors
ncodes – size of the binary codes (bytes)
ordered – if != 0: order the results by decreasing distance (may be bottleneck for k/n > 0.01)
-
void hammings_knn(int_maxheap_array_t *ha, const uint8_t *a, const uint8_t *b, size_t nb, size_t ncodes, int ordered)
-
void hammings_knn_mc(const uint8_t *a, const uint8_t *b, size_t na, size_t nb, size_t k, size_t ncodes, int32_t *distances, int64_t *labels)
Return the k smallest Hamming distances for a set of binary query vectors, using counting max.
- Parameters:
a – queries, size na * ncodes
b – database, size nb * ncodes
na – number of query vectors
nb – number of database vectors
k – number of vectors/distances to return
ncodes – size of the binary codes (bytes)
distances – output distances from each query vector to its k nearest neighbors
labels – output ids of the k nearest neighbors to each query vector
-
void hamming_range_search(const uint8_t *a, const uint8_t *b, size_t na, size_t nb, int radius, size_t ncodes, RangeSearchResult *result)
same as hammings_knn except we are doing a range search with radius
-
void hamming_count_thres(const uint8_t *bs1, const uint8_t *bs2, size_t n1, size_t n2, hamdis_t ht, size_t ncodes, size_t *nptr)
-
size_t match_hamming_thres(const uint8_t *bs1, const uint8_t *bs2, size_t n1, size_t n2, hamdis_t ht, size_t ncodes, int64_t *idx, hamdis_t *dis)
-
void crosshamming_count_thres(const uint8_t *dbs, size_t n, hamdis_t ht, size_t ncodes, size_t *nptr)
-
hamdis_t hamming(const uint64_t *bs1, const uint64_t *bs2, size_t nwords)
-
template<class C>
inline void heap_pop(size_t k, typename C::T *bh_val, typename C::TI *bh_ids) Pops the top element from the heap defined by bh_val[0..k-1] and bh_ids[0..k-1]. on output the element at k-1 is undefined.
-
template<class C>
inline void heap_push(size_t k, typename C::T *bh_val, typename C::TI *bh_ids, typename C::T val, typename C::TI id) Pushes the element (val, ids) into the heap bh_val[0..k-2] and bh_ids[0..k-2]. on output the element at k-1 is defined.
-
template<class C>
inline void heap_replace_top(size_t k, typename C::T *bh_val, typename C::TI *bh_ids, typename C::T val, typename C::TI id) Replaces the top element from the heap defined by bh_val[0..k-1] and bh_ids[0..k-1], and for identical bh_val[] values also sorts by bh_ids[] values.
-
template<typename T>
inline void minheap_pop(size_t k, T *bh_val, int64_t *bh_ids)
-
template<typename T>
inline void minheap_push(size_t k, T *bh_val, int64_t *bh_ids, T val, int64_t ids)
-
template<typename T>
inline void minheap_replace_top(size_t k, T *bh_val, int64_t *bh_ids, T val, int64_t ids)
-
template<typename T>
inline void maxheap_pop(size_t k, T *bh_val, int64_t *bh_ids)
-
template<typename T>
inline void maxheap_push(size_t k, T *bh_val, int64_t *bh_ids, T val, int64_t ids)
-
template<typename T>
inline void maxheap_replace_top(size_t k, T *bh_val, int64_t *bh_ids, T val, int64_t ids)
-
template<class C>
inline void heap_heapify(size_t k, typename C::T *bh_val, typename C::TI *bh_ids, const typename C::T *x = nullptr, const typename C::TI *ids = nullptr, size_t k0 = 0)
-
template<typename T>
inline void minheap_heapify(size_t k, T *bh_val, int64_t *bh_ids, const T *x = nullptr, const int64_t *ids = nullptr, size_t k0 = 0)
-
template<typename T>
inline void maxheap_heapify(size_t k, T *bh_val, int64_t *bh_ids, const T *x = nullptr, const int64_t *ids = nullptr, size_t k0 = 0)
-
template<class C>
inline void heap_addn(size_t k, typename C::T *bh_val, typename C::TI *bh_ids, const typename C::T *x, const typename C::TI *ids, size_t n)
-
template<typename T>
inline void minheap_addn(size_t k, T *bh_val, int64_t *bh_ids, const T *x, const int64_t *ids, size_t n)
-
template<typename T>
inline void maxheap_addn(size_t k, T *bh_val, int64_t *bh_ids, const T *x, const int64_t *ids, size_t n)
-
template<typename C>
inline size_t heap_reorder(size_t k, typename C::T *bh_val, typename C::TI *bh_ids)
-
template<typename T>
inline size_t minheap_reorder(size_t k, T *bh_val, int64_t *bh_ids)
-
template<typename T>
inline size_t maxheap_reorder(size_t k, T *bh_val, int64_t *bh_ids)
-
template<class C>
inline void indirect_heap_pop(size_t k, const typename C::T *bh_val, typename C::TI *bh_ids)
-
template<class C>
inline void indirect_heap_push(size_t k, const typename C::T *bh_val, typename C::TI *bh_ids, typename C::TI id)
-
template<>
inline float cmin_nextafter<float>(float x)
-
template<>
inline float cmax_nextafter<float>(float x)
-
template<>
inline uint16_t cmin_nextafter<uint16_t>(uint16_t x)
-
template<>
inline uint16_t cmax_nextafter<uint16_t>(uint16_t x)
-
template<class C>
C::T partition_fuzzy(typename C::T *vals, typename C::TI *ids, size_t n, size_t q_min, size_t q_max, size_t *q_out) partitions the table into 0:q and q:n where all elements above q are >= all elements below q (for C = CMax, for CMin comparisons are reversed)
Returns the partition threshold. The elements q:n are destroyed on output.
-
template<class C>
inline C::T partition(typename C::T *vals, typename C::TI *ids, size_t n, size_t q) simplified interface for when the parition is not fuzzy
-
void simd_histogram_8(const uint16_t *data, int n, uint16_t min, int shift, int *hist)
low level SIMD histogramming functions 8-bin histogram of (x - min) >> shift values outside the range are ignored. the data table should be aligned on 32 bytes
-
void simd_histogram_16(const uint16_t *data, int n, uint16_t min, int shift, int *hist)
same for 16-bin histogram
-
void float_rand(float *x, size_t n, int64_t seed)
-
void float_randn(float *x, size_t n, int64_t seed)
-
void int64_rand(int64_t *x, size_t n, int64_t seed)
-
void byte_rand(uint8_t *x, size_t n, int64_t seed)
-
void int64_rand_max(int64_t *x, size_t n, uint64_t max, int64_t seed)
-
void rand_perm(int *perm, size_t n, int64_t seed)
-
void rand_smooth_vectors(size_t n, size_t d, float *x, int64_t seed)
-
inline simd16uint16 min(simd16uint16 a, simd16uint16 b)
-
inline simd16uint16 max(simd16uint16 a, simd16uint16 b)
-
inline simd16uint16 combine2x2(simd16uint16 a, simd16uint16 b)
-
inline uint32_t cmp_ge32(simd16uint16 d0, simd16uint16 d1, simd16uint16 thr)
-
inline uint32_t cmp_le32(simd16uint16 d0, simd16uint16 d1, simd16uint16 thr)
-
inline simd32uint8 uint16_to_uint8_saturate(simd16uint16 a, simd16uint16 b)
-
inline uint32_t get_MSBs(simd32uint8 a)
get most significant bit of each byte
-
inline simd32uint8 blendv(simd32uint8 a, simd32uint8 b, simd32uint8 mask)
use MSB of each byte of mask to select a byte between a and b
-
inline simd8float32 hadd(simd8float32 a, simd8float32 b)
-
inline simd8float32 unpacklo(simd8float32 a, simd8float32 b)
-
inline simd8float32 unpackhi(simd8float32 a, simd8float32 b)
-
inline simd8float32 fmadd(simd8float32 a, simd8float32 b, simd8float32 c)
-
inline simd16uint16 min(const simd16uint16 &av, const simd16uint16 &bv)
-
inline simd16uint16 max(const simd16uint16 &av, const simd16uint16 &bv)
-
inline simd16uint16 combine2x2(const simd16uint16 &a, const simd16uint16 &b)
-
inline uint32_t cmp_ge32(const simd16uint16 &d0, const simd16uint16 &d1, const simd16uint16 &thr)
-
inline uint32_t cmp_le32(const simd16uint16 &d0, const simd16uint16 &d1, const simd16uint16 &thr)
-
inline simd32uint8 uint16_to_uint8_saturate(const simd16uint16 &a, const simd16uint16 &b)
-
inline uint32_t get_MSBs(const simd32uint8 &a)
get most significant bit of each byte
-
inline simd32uint8 blendv(const simd32uint8 &a, const simd32uint8 &b, const simd32uint8 &mask)
use MSB of each byte of mask to select a byte between a and b
-
inline simd8float32 hadd(const simd8float32 &a, const simd8float32 &b)
-
inline simd8float32 unpacklo(const simd8float32 &a, const simd8float32 &b)
-
inline simd8float32 unpackhi(const simd8float32 &a, const simd8float32 &b)
-
inline simd8float32 fmadd(const simd8float32 &a, const simd8float32 &b, const simd8float32 &c)
-
std::string get_compile_options()
get compile options
-
double getmillisecs()
ms elapsed since some arbitrary epoch
-
size_t get_mem_usage_kb()
get current RSS usage in kB
-
uint64_t get_cycles()
-
void fvec_madd(size_t n, const float *a, float bf, const float *b, float *c)
compute c := a + bf * b for a, b and c tables
- Parameters:
n – size of the tables
a – size n
b – size n
c – restult table, size n
-
int fvec_madd_and_argmin(size_t n, const float *a, float bf, const float *b, float *c)
same as fvec_madd, also return index of the min of the result table
- Returns:
index of the min of table c
-
void reflection(const float *u, float *x, size_t n, size_t d, size_t nu)
-
void matrix_qr(int m, int n, float *a)
compute the Q of the QR decomposition for m > n
- Parameters:
a – size n * m: input matrix and output Q
-
void ranklist_handle_ties(int k, int64_t *idx, const float *dis)
distances are supposed to be sorted. Sorts indices with same distance
-
size_t ranklist_intersection_size(size_t k1, const int64_t *v1, size_t k2, const int64_t *v2)
count the number of common elements between v1 and v2 algorithm = sorting + bissection to avoid double-counting duplicates
-
size_t merge_result_table_with(size_t n, size_t k, int64_t *I0, float *D0, const int64_t *I1, const float *D1, bool keep_min = true, int64_t translation = 0)
merge a result table into another one
- Parameters:
I0, D0 – first result table, size (n, k)
I1, D1 – second result table, size (n, k)
keep_min – if true, keep min values, otherwise keep max
translation – add this value to all I1’s indexes
- Returns:
nb of values that were taken from the second table
-
double imbalance_factor(int n, int k, const int64_t *assign)
a balanced assignment has a IF of 1
-
double imbalance_factor(int k, const int *hist)
same, takes a histogram as input
-
void fvec_argsort(size_t n, const float *vals, size_t *perm)
-
void fvec_argsort_parallel(size_t n, const float *vals, size_t *perm)
-
int ivec_hist(size_t n, const int *v, int vmax, int *hist)
compute histogram on v
-
void bincode_hist(size_t n, size_t nbits, const uint8_t *codes, int *hist)
Compute histogram of bits on a code array
- Parameters:
codes – size(n, nbits / 8)
hist – size(nbits): nb of 1s in the array of codes
-
size_t ivec_checksum(size_t n, const int *a)
compute a checksum on a table.
-
const float *fvecs_maybe_subsample(size_t d, size_t *n, size_t nmax, const float *x, bool verbose = false, int64_t seed = 1234)
random subsamples a set of vectors if there are too many of them
- Parameters:
d – dimension of the vectors
n – on input: nb of input vectors, output: nb of output vectors
nmax – max nb of vectors to keep
x – input array, size *n-by-d
seed – random seed to use for sampling
- Returns:
x or an array allocated with new [] with *n vectors
-
void binary_to_real(size_t d, const uint8_t *x_in, float *x_out)
Convert binary vector to +1/-1 valued float vector.
- Parameters:
d – dimension of the vector (multiple of 8)
x_in – input binary vector (uint8_t table of size d / 8)
x_out – output float vector (float table of size d)
-
void real_to_binary(size_t d, const float *x_in, uint8_t *x_out)
Convert float vector to binary vector. Components > 0 are converted to 1, others to 0.
- Parameters:
d – dimension of the vector (multiple of 8)
x_in – input float vector (float table of size d)
x_out – output binary vector (uint8_t table of size d / 8)
-
uint64_t hash_bytes(const uint8_t *bytes, int64_t n)
A reasonable hashing function
-
bool check_openmp()
Whether OpenMP annotations were respected.
Variables
- FAISS_API int index2layer_sa_encode_bs
- FAISS_API int index_factory_verbose
set to > 0 to get more logs from index_factory
-
const int IO_FLAG_READ_ONLY = 2
-
const int IO_FLAG_ONDISK_SAME_DIR = 4
-
const int IO_FLAG_SKIP_IVF_DATA = 8
-
const int IO_FLAG_MMAP = IO_FLAG_SKIP_IVF_DATA | 0x646f0000
- FAISS_API IndexBinaryHashStats indexBinaryHash_stats
- FAISS_API FastScanStats FastScan_stats
- FAISS_API IndexIVFStats indexIVF_stats
- FAISS_API IVFFastScanStats IVFFastScan_stats
- FAISS_API size_t precomputed_table_max_bytes
- FAISS_API int index_ivfpq_add_core_o_bs
- FAISS_API IndexIVFPQStats indexIVFPQ_stats
- FAISS_API IndexPQStats indexPQ_stats
- FAISS_API int multi_index_quantizer_search_bs
- FAISS_API int distance_compute_blas_threshold
- FAISS_API int distance_compute_blas_query_bs
- FAISS_API int distance_compute_blas_database_bs
- FAISS_API int distance_compute_min_k_reservoir
-
const uint8_t hamdis_tab_ham_bytes[256]
- FAISS_API size_t hamming_batch_size
- FAISS_API PartitionStats partition_stats
-
struct AdditiveCoarseQuantizer : public faiss::Index
- #include <IndexAdditiveQuantizer.h>
A “virtual” index where the elements are the residual quantizer centroids.
Intended for use as a coarse quantizer in an IndexIVF.
Subclassed by faiss::LocalSearchCoarseQuantizer, faiss::ResidualCoarseQuantizer
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
explicit AdditiveCoarseQuantizer(idx_t d = 0, AdditiveQuantizer *aq = nullptr, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
N/A.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void reset() override
N/A.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
AdditiveQuantizer *aq
-
std::vector<float> centroid_norms
norms of centroids, useful for knn-search
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
template<class T, int A = 32>
struct AlignedTable Public Functions
-
inline AlignedTable()
-
inline explicit AlignedTable(size_t n)
-
inline size_t itemsize() const
-
inline void resize(size_t n)
-
inline void clear()
-
inline size_t size() const
-
inline size_t nbytes() const
-
inline T *get()
-
inline const T *get() const
-
inline T *data()
-
inline const T *data() const
-
inline T &operator[](size_t i)
-
inline T operator[](size_t i) const
Public Static Functions
-
static inline size_t round_capacity(size_t n)
-
inline AlignedTable()
-
template<class T, int A = 32>
struct AlignedTableTightAlloc Public Functions
-
inline AlignedTableTightAlloc()
-
inline explicit AlignedTableTightAlloc(size_t n)
-
inline size_t itemsize() const
-
inline void resize(size_t n)
-
inline void clear()
-
inline size_t size() const
-
inline size_t nbytes() const
-
inline T *get()
-
inline const T *get() const
-
inline T *data()
-
inline const T *data() const
-
inline T &operator[](size_t i)
-
inline T operator[](size_t i) const
-
inline ~AlignedTableTightAlloc()
-
inline AlignedTableTightAlloc<T, A> &operator=(const AlignedTableTightAlloc<T, A> &other)
-
inline AlignedTableTightAlloc(const AlignedTableTightAlloc<T, A> &other)
-
inline AlignedTableTightAlloc()
-
struct ArrayInvertedLists : public faiss::InvertedLists
- #include <InvertedLists.h>
simple (default) implementation as an array of inverted lists
Public Functions
-
ArrayInvertedLists(size_t nlist, size_t code_size)
-
virtual size_t list_size(size_t list_no) const override
get the size of a list
-
virtual const uint8_t *get_codes(size_t list_no) const override
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const override
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void resize(size_t list_no, size_t new_size) override
-
~ArrayInvertedLists() override
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const
release codes returned by get_codes (default implementation is nop
-
virtual void release_ids(size_t list_no, const idx_t *ids) const
release ids returned by get_ids
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code)
add one entry to an inverted list
-
virtual void update_entry(size_t list_no, size_t offset, idx_t id, const uint8_t *code)
-
virtual void reset()
-
void merge_from(InvertedLists *oivf, size_t add_id)
move all entries from oivf (empty on output)
-
double imbalance_factor() const
1= perfectly balanced, >1: imbalanced
-
void print_stats() const
display some stats about the inverted lists
-
size_t compute_ntotal() const
sum up list sizes
Public Members
-
std::vector<std::vector<uint8_t>> codes
-
std::vector<std::vector<idx_t>> ids
Inverted lists for indexes.
-
size_t nlist
number of possible key values
-
size_t code_size
code size per vector in bytes
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
ArrayInvertedLists(size_t nlist, size_t code_size)
-
struct AutoTuneCriterion
- #include <AutoTune.h>
Evaluation criterion. Returns a performance measure in [0,1], higher is better.
Subclassed by faiss::IntersectionCriterion, faiss::OneRecallAtRCriterion
Public Functions
-
void set_groundtruth(int gt_nnn, const float *gt_D_in, const idx_t *gt_I_in)
Intitializes the gt_D and gt_I vectors. Must be called before evaluating
- Parameters:
gt_D_in – size nq * gt_nnn
gt_I_in – size nq * gt_nnn
-
virtual double evaluate(const float *D, const idx_t *I) const = 0
Evaluate the criterion.
- Parameters:
D – size nq * nnn
I – size nq * nnn
- Returns:
the criterion, between 0 and 1. Larger is better.
-
inline virtual ~AutoTuneCriterion()
Public Members
-
idx_t nq
nb of queries this criterion is evaluated on
-
idx_t nnn
nb of NNs that the query should request
-
idx_t gt_nnn
nb of GT NNs required to evaluate criterion
-
std::vector<float> gt_D
Ground-truth distances (size nq * gt_nnn)
-
std::vector<idx_t> gt_I
Ground-truth indexes (size nq * gt_nnn)
-
void set_groundtruth(int gt_nnn, const float *gt_D_in, const idx_t *gt_I_in)
-
struct BinaryInvertedListScanner
-
Public Functions
-
virtual void set_query(const uint8_t *query_vector) = 0
from now on we handle this query.
-
virtual void set_list(idx_t list_no, uint8_t coarse_dis) = 0
following codes come from this inverted list
-
virtual uint32_t distance_to_code(const uint8_t *code) const = 0
compute a single query-to-code distance
-
virtual size_t scan_codes(size_t n, const uint8_t *codes, const idx_t *ids, int32_t *distances, idx_t *labels, size_t k) const = 0
compute the distances to codes. (distances, labels) should be organized as a min- or max-heap
- Parameters:
n – number of codes to scan
codes – codes to scan (n * code_size)
ids – corresponding ids (ignored if store_pairs)
distances – heap distances (size k)
labels – heap labels (size k)
k – heap size
-
virtual void scan_codes_range(size_t n, const uint8_t *codes, const idx_t *ids, int radius, RangeQueryResult &result) const = 0
-
inline virtual ~BinaryInvertedListScanner()
-
virtual void set_query(const uint8_t *query_vector) = 0
-
struct BitstringReader
Public Functions
-
inline BitstringReader(const uint8_t *code, size_t code_size)
-
inline uint64_t read(int nbit)
Public Members
-
const uint8_t *code
-
size_t code_size
-
size_t i
-
inline BitstringReader(const uint8_t *code, size_t code_size)
-
struct BitstringWriter
Public Functions
-
inline BitstringWriter(uint8_t *code, size_t code_size)
-
inline void write(uint64_t x, int nbit)
Public Members
-
uint8_t *code
-
size_t code_size
-
size_t i
-
inline BitstringWriter(uint8_t *code, size_t code_size)
-
struct BlockInvertedLists : public faiss::InvertedLists
- #include <BlockInvertedLists.h>
Inverted Lists that are organized by blocks.
Different from the regular inverted lists, the codes are organized by blocks of size block_size bytes that reprsent a set of n_per_block. Therefore, code allocations are always rounded up to block_size bytes. The codes are also aligned on 32-byte boundaries for use with SIMD.
To avoid misinterpretations, the code_size is set to (size_t)(-1), even if arguably the amount of memory consumed by code is block_size / n_per_block.
The writing functions add_entries and update_entries operate on block-aligned data.
Public Functions
-
BlockInvertedLists(size_t nlist, size_t vec_per_block, size_t block_size)
-
BlockInvertedLists()
-
virtual size_t list_size(size_t list_no) const override
get the size of a list
-
virtual const uint8_t *get_codes(size_t list_no) const override
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const override
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) override
not implemented
-
virtual void resize(size_t list_no, size_t new_size) override
-
~BlockInvertedLists() override
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const¶
release codes returned by get_codes (default implementation is nop
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const¶
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const¶
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const¶
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code)¶
add one entry to an inverted list
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) = 0
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) = 0
-
virtual void reset()¶
-
void merge_from(InvertedLists *oivf, size_t add_id)¶
move all entries from oivf (empty on output)
-
double imbalance_factor() const¶
1= perfectly balanced, >1: imbalanced
-
void print_stats() const¶
display some stats about the inverted lists
-
size_t compute_ntotal() const¶
sum up list sizes
Public Members
-
size_t n_per_block
-
size_t block_size
-
std::vector<AlignedTable<uint8_t>> codes
-
std::vector<std::vector<idx_t>> ids
-
size_t nlist¶
number of possible key values
-
size_t code_size¶
code size per vector in bytes
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)¶
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
BlockInvertedLists(size_t nlist, size_t vec_per_block, size_t block_size)
-
struct BlockInvertedListsIOHook : public faiss::InvertedListsIOHook
Public Functions
-
BlockInvertedListsIOHook()
-
virtual void write(const InvertedLists *ils, IOWriter *f) const override
write the index to the IOWriter (including the fourcc)
-
virtual InvertedLists *read(IOReader *f, int io_flags) const override
called when the fourcc matches this class’s fourcc
-
virtual InvertedLists *read_ArrayInvertedLists(IOReader *f, int io_flags, size_t nlist, size_t code_size, const std::vector<size_t> &sizes) const¶
read from a ArrayInvertedLists into this invertedlist type. For this to work, the callback has to be enabled and the io_flag has to be set to IO_FLAG_SKIP_IVF_DATA | (16 upper bits of the fourcc)
(default implementation fails)
Public Members
-
const std::string key¶
string version of the fourcc
-
const std::string classname¶
typeid.name
Public Static Functions
-
static void add_callback(InvertedListsIOHook*)¶
-
static void print_callbacks()¶
-
static InvertedListsIOHook *lookup(int h)¶
-
static InvertedListsIOHook *lookup_classname(const std::string &classname)¶
-
BlockInvertedListsIOHook()
-
struct CenteringTransform : public faiss::VectorTransform
- #include <VectorTransform.h>
Subtract the mean of each component from the vectors.
Public Functions
-
explicit CenteringTransform(int d = 0)
-
virtual void train(idx_t n, const float *x) override
train on n vectors.
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
subtract the mean
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
add the mean
-
float *apply(idx_t n, const float *x) const
apply the random rotation, return new allocated matrix
- Parameters:
x – size n * d_in
- Returns:
size n * d_out
Public Members
-
std::vector<float> mean
Mean, size d_in = d_out.
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit CenteringTransform(int d = 0)
-
struct Cloner
- #include <clone_index.h>
Cloner class, useful to override classes with other cloning functions. The cloning function above just calls Cloner::clone_Index.
Subclassed by faiss::gpu::ToCPUCloner, faiss::gpu::ToGpuCloner, faiss::gpu::ToGpuClonerMultiple
Public Functions
-
virtual VectorTransform *clone_VectorTransform(const VectorTransform*)
-
inline virtual ~Cloner()
-
virtual VectorTransform *clone_VectorTransform(const VectorTransform*)
-
struct Clustering : public faiss::ClusteringParameters
- #include <Clustering.h>
K-means clustering based on assignment - centroid update iterations
The clustering is based on an Index object that assigns training points to the centroids. Therefore, at each iteration the centroids are added to the index.
On output, the centoids table is set to the latest version of the centroids and they are also added to the index. If the centroids table it is not empty on input, it is also used for initialization.
Subclassed by faiss::Clustering1D
Public Functions
-
Clustering(int d, int k)
-
Clustering(int d, int k, const ClusteringParameters &cp)
-
virtual void train(idx_t n, const float *x, faiss::Index &index, const float *x_weights = nullptr)
run k-means training
- Parameters:
x – training vectors, size n * d
index – index used for assignment
x_weights – weight associated to each vector: NULL or size n
-
void train_encoded(idx_t nx, const uint8_t *x_in, const Index *codec, Index &index, const float *weights = nullptr)
run with encoded vectors
win addition to train()’s parameters takes a codec as parameter to decode the input vectors.
- Parameters:
codec – codec used to decode the vectors (nullptr = vectors are in fact floats) *
-
void post_process_centroids()
Post-process the centroids after each centroid update. includes optional L2 normalization and nearest integer rounding
-
inline virtual ~Clustering()
Public Members
-
size_t d
dimension of the vectors
-
size_t k
nb of centroids
-
std::vector<float> centroids
centroids (k * d) if centroids are set on input to train, they will be used as initialization
-
std::vector<ClusteringIterationStats> iteration_stats
stats at every iteration of clustering
-
int niter
clustering iterations
-
int nredo
redo clustering this many times and keep best
-
bool verbose
-
bool spherical
do we want normalized centroids?
-
bool int_centroids
round centroids coordinates to integer
-
bool update_index
re-train index after each iteration?
-
bool frozen_centroids
use the centroids provided as input and do not change them during iterations
-
int min_points_per_centroid
otherwise you get a warning
-
int max_points_per_centroid
to limit size of dataset
-
int seed
seed for the random number generator
-
size_t decode_block_size
how many vectors at a time to decode
-
Clustering(int d, int k)
-
struct Clustering1D : public faiss::Clustering
- #include <Clustering.h>
Exact 1D clustering algorithm
Since it does not use an index, it does not overload the train() function
Public Functions
-
explicit Clustering1D(int k)
-
Clustering1D(int k, const ClusteringParameters &cp)
-
void train_exact(idx_t n, const float *x)
-
inline virtual ~Clustering1D()
-
virtual void train(idx_t n, const float *x, faiss::Index &index, const float *x_weights = nullptr)
run k-means training
- Parameters:
x – training vectors, size n * d
index – index used for assignment
x_weights – weight associated to each vector: NULL or size n
-
void train_encoded(idx_t nx, const uint8_t *x_in, const Index *codec, Index &index, const float *weights = nullptr)
run with encoded vectors
win addition to train()’s parameters takes a codec as parameter to decode the input vectors.
- Parameters:
codec – codec used to decode the vectors (nullptr = vectors are in fact floats) *
-
void post_process_centroids()
Post-process the centroids after each centroid update. includes optional L2 normalization and nearest integer rounding
Public Members
-
size_t d
dimension of the vectors
-
size_t k
nb of centroids
-
std::vector<float> centroids
centroids (k * d) if centroids are set on input to train, they will be used as initialization
-
std::vector<ClusteringIterationStats> iteration_stats
stats at every iteration of clustering
-
int niter
clustering iterations
-
int nredo
redo clustering this many times and keep best
-
bool verbose
-
bool spherical
do we want normalized centroids?
-
bool int_centroids
round centroids coordinates to integer
-
bool update_index
re-train index after each iteration?
-
bool frozen_centroids
use the centroids provided as input and do not change them during iterations
-
int min_points_per_centroid
otherwise you get a warning
-
int max_points_per_centroid
to limit size of dataset
-
int seed
seed for the random number generator
-
size_t decode_block_size
how many vectors at a time to decode
-
explicit Clustering1D(int k)
-
struct ClusteringIterationStats
Public Members
-
float obj
objective values (sum of distances reported by index)
-
double time
seconds for iteration
-
double time_search
seconds for just search
-
double imbalance_factor
imbalance factor of iteration
-
int nsplit
number of cluster splits
-
float obj
-
struct ClusteringParameters
- #include <Clustering.h>
Class for the clustering parameters. Can be passed to the constructor of the Clustering object.
Subclassed by faiss::Clustering, faiss::ProgressiveDimClusteringParameters
Public Functions
-
ClusteringParameters()
sets reasonable defaults
Public Members
-
int niter
clustering iterations
-
int nredo
redo clustering this many times and keep best
-
bool verbose
-
bool spherical
do we want normalized centroids?
-
bool int_centroids
round centroids coordinates to integer
-
bool update_index
re-train index after each iteration?
-
bool frozen_centroids
use the centroids provided as input and do not change them during iterations
-
int min_points_per_centroid
otherwise you get a warning
-
int max_points_per_centroid
to limit size of dataset
-
int seed
seed for the random number generator
-
size_t decode_block_size
how many vectors at a time to decode
-
ClusteringParameters()
-
template<typename T_, typename TI_>
struct CMax - #include <ordered_key_value.h>
The C object gives the type T of the values of a key-value storage, the type of the keys, TI and the comparison that is done: CMax for a decreasing series and CMin for increasing series. In other words, for a given threshold threshold, an incoming value x is kept if
C::cmp(threshold, x)
is true.
Public Static Functions
-
static inline T neutral()
Public Static Attributes
-
static const bool is_max = true
-
static inline T neutral()
-
template<typename T_, typename TI_>
struct CMin -
Public Static Functions
-
static inline T neutral()
Public Static Attributes
-
static const bool is_max = false
-
static inline T neutral()
-
struct DirectMap
- #include <DirectMap.h>
Direct map: a way to map back from ids to inverted lists
Public Types
-
enum Type
Values:
-
enumerator NoMap
-
enumerator Array
-
enumerator Hashtable
-
enumerator NoMap
Public Functions
-
DirectMap()
-
void set_type(Type new_type, const InvertedLists *invlists, size_t ntotal)
set type and initialize
-
inline bool no() const
for quick checks
-
void check_can_add(const idx_t *ids)
throw if Array and ids is not NULL
update the direct_map
-
void clear()
remove all entries
-
size_t remove_ids(const IDSelector &sel, InvertedLists *invlists)
remove ids from the InvertedLists, possibly using the direct map
operations on inverted lists that require translation with a DirectMap
-
void update_codes(InvertedLists *invlists, int n, const idx_t *ids, const idx_t *list_nos, const uint8_t *codes)
update entries, using the direct map
-
enum Type
-
struct DirectMapAdd
- #include <DirectMap.h>
Thread-safe way of updating the direct_map.
Public Functions
-
void add(size_t i, idx_t list_no, size_t offset)
add vector i (with id xids[i]) at list_no and offset
-
~DirectMapAdd()
-
void add(size_t i, idx_t list_no, size_t offset)
-
struct FastScanStats
Public Functions
-
inline FastScanStats()
-
inline void reset()
Public Members
-
uint64_t t0
-
uint64_t t1
-
uint64_t t2
-
uint64_t t3
-
inline FastScanStats()
-
struct GenHammingComputer16
Public Functions
-
inline GenHammingComputer16(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
Public Members
-
uint64_t a0
-
uint64_t a1
-
inline GenHammingComputer16(const uint8_t *a8, int code_size)
-
struct GenHammingComputer32
Public Functions
-
inline GenHammingComputer32(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
Public Members
-
uint64_t a0
-
uint64_t a1
-
uint64_t a2
-
uint64_t a3
-
inline GenHammingComputer32(const uint8_t *a8, int code_size)
-
struct GenHammingComputer8
Public Functions
-
inline GenHammingComputer8(const uint8_t *a, int code_size)
-
inline int hamming(const uint8_t *b) const
Public Members
-
uint64_t a0
-
inline GenHammingComputer8(const uint8_t *a, int code_size)
-
struct GenHammingComputerM8
Public Functions
-
inline GenHammingComputerM8(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
Public Members
-
const uint64_t *a
-
int n
-
inline GenHammingComputerM8(const uint8_t *a8, int code_size)
-
template<int CODE_SIZE>
struct HammingComputer : public faiss::HammingComputerDefault Public Functions
-
inline HammingComputer(const uint8_t *a, int code_size)
-
inline void set(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
Public Members
-
const uint8_t *a8
-
int quotient8
-
int remainder8
-
inline HammingComputer(const uint8_t *a, int code_size)
-
struct HammingComputer16
Public Functions
-
inline HammingComputer16()
-
inline HammingComputer16(const uint8_t *a8, int code_size)
-
inline void set(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
Public Members
-
uint64_t a0
-
uint64_t a1
-
inline HammingComputer16()
-
struct HammingComputer20
Public Functions
-
inline HammingComputer20()
-
inline HammingComputer20(const uint8_t *a8, int code_size)
-
inline void set(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
Public Members
-
uint64_t a0
-
uint64_t a1
-
uint32_t a2
-
inline HammingComputer20()
-
struct HammingComputer32
Public Functions
-
inline HammingComputer32()
-
inline HammingComputer32(const uint8_t *a8, int code_size)
-
inline void set(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
Public Members
-
uint64_t a0
-
uint64_t a1
-
uint64_t a2
-
uint64_t a3
-
inline HammingComputer32()
-
struct HammingComputer4
Public Functions
-
inline HammingComputer4()
-
inline HammingComputer4(const uint8_t *a, int code_size)
-
inline void set(const uint8_t *a, int code_size)
-
inline int hamming(const uint8_t *b) const
Public Members
-
uint32_t a0
-
inline HammingComputer4()
-
struct HammingComputer64
Public Functions
-
inline HammingComputer64()
-
inline HammingComputer64(const uint8_t *a8, int code_size)
-
inline void set(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
Public Members
-
uint64_t a0
-
uint64_t a1
-
uint64_t a2
-
uint64_t a3
-
uint64_t a4
-
uint64_t a5
-
uint64_t a6
-
uint64_t a7
-
inline HammingComputer64()
-
struct HammingComputer8
Public Functions
-
inline HammingComputer8()
-
inline HammingComputer8(const uint8_t *a, int code_size)
-
inline void set(const uint8_t *a, int code_size)
-
inline int hamming(const uint8_t *b) const
Public Members
-
uint64_t a0
-
inline HammingComputer8()
-
struct HammingComputerDefault
Subclassed by faiss::HammingComputer< CODE_SIZE >
Public Functions
-
inline HammingComputerDefault()
-
inline HammingComputerDefault(const uint8_t *a8, int code_size)
-
inline void set(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
Public Members
-
const uint8_t *a8
-
int quotient8
-
int remainder8
-
inline HammingComputerDefault()
-
struct HammingComputerM4
Public Functions
-
inline HammingComputerM4()
-
inline HammingComputerM4(const uint8_t *a4, int code_size)
-
inline void set(const uint8_t *a4, int code_size)
-
inline int hamming(const uint8_t *b8) const
Public Members
-
const uint32_t *a
-
int n
-
inline HammingComputerM4()
-
struct HammingComputerM8
Public Functions
-
inline HammingComputerM8()
-
inline HammingComputerM8(const uint8_t *a8, int code_size)
-
inline void set(const uint8_t *a8, int code_size)
-
inline int hamming(const uint8_t *b8) const
Public Members
-
const uint64_t *a
-
int n
-
inline HammingComputerM8()
-
template<class HammingComputer>
struct HCounterState - #include <hamming-inl.h>
This class maintains a list of best distances seen so far.
Since the distances are in a limited range (0 to nbit), the object maintains one list per possible distance, and fills in only the n-first lists, such that the sum of sizes of the n lists is below k.
Public Functions
-
inline HCounterState(int *counters, int64_t *ids_per_dis, const uint8_t *x, int d, int k)
-
inline void update_counter(const uint8_t *y, size_t j)
Public Members
-
int *counters
-
int64_t *ids_per_dis
-
int thres
-
int count_lt
-
int count_eq
-
int k
-
inline HCounterState(int *counters, int64_t *ids_per_dis, const uint8_t *x, int d, int k)
-
template<typename C>
struct HeapArray - #include <Heap.h>
a template structure for a set of [min|max]-heaps it is tailored so that the actual data of the heaps can just live in compact arrays.
Public Functions
-
inline T *get_val(size_t key)
Return the list of values for a heap.
-
inline TI *get_ids(size_t key)
Correspponding identifiers.
-
void heapify()
prepare all the heaps before adding
-
void addn(size_t nj, const T *vin, TI j0 = 0, size_t i0 = 0, int64_t ni = -1)
add nj elements to heaps i0:i0+ni, with sequential ids
- Parameters:
nj – nb of elements to add to each heap
vin – elements to add, size ni * nj
j0 – add this to the ids that are added
i0 – first heap to update
ni – nb of elements to update (-1 = use nh)
-
void addn_with_ids(size_t nj, const T *vin, const TI *id_in = nullptr, int64_t id_stride = 0, size_t i0 = 0, int64_t ni = -1)
same as addn
- Parameters:
id_in – ids of the elements to add, size ni * nj
id_stride – stride for id_in
-
void reorder()
reorder all the heaps
-
inline T *get_val(size_t key)
-
struct HStackInvertedLists : public faiss::ReadOnlyInvertedLists
- #include <InvertedLists.h>
Horizontal stack of inverted lists.
Public Functions
-
HStackInvertedLists(int nil, const InvertedLists **ils)
build InvertedLists by concatenating nil of them
-
virtual size_t list_size(size_t list_no) const override
get the size of a list
-
virtual const uint8_t *get_codes(size_t list_no) const override
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const override
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const override
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const override
release codes returned by get_codes (default implementation is nop
-
virtual void release_ids(size_t list_no, const idx_t *ids) const override
release ids returned by get_ids
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const override
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const override
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void resize(size_t list_no, size_t new_size) override
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code)
add one entry to an inverted list
-
virtual void update_entry(size_t list_no, size_t offset, idx_t id, const uint8_t *code)
-
virtual void reset()
-
void merge_from(InvertedLists *oivf, size_t add_id)
move all entries from oivf (empty on output)
-
double imbalance_factor() const
1= perfectly balanced, >1: imbalanced
-
void print_stats() const
display some stats about the inverted lists
-
size_t compute_ntotal() const
sum up list sizes
Public Members
-
std::vector<const InvertedLists*> ils
-
size_t nlist
number of possible key values
-
size_t code_size
code size per vector in bytes
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
HStackInvertedLists(int nil, const InvertedLists **ils)
-
struct Index
- #include <Index.h>
Abstract structure for an index, supports adding vectors and searching them.
All vectors provided at add or search time are 32-bit float arrays, although the internal representation may vary.
Subclassed by faiss::AdditiveCoarseQuantizer, faiss::gpu::GpuIndex, faiss::IndexFastScan, faiss::IndexFlatCodes, faiss::IndexHNSW, faiss::IndexIVF, faiss::IndexLattice, faiss::IndexNNDescent, faiss::IndexNSG, faiss::IndexPreTransform, faiss::IndexRefine, faiss::IndexSplitVectors, faiss::MultiIndexQuantizer
Public Types
-
using idx_t = int64_t
all indices are this type
-
using component_t = float
-
using distance_t = float
Public Functions
-
inline explicit Index(idx_t d = 0, MetricType metric = METRIC_L2)
-
virtual ~Index()
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add(idx_t n, const float *x) = 0
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const = 0
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reset() = 0
removes all elements from the database.
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
-
using idx_t = int64_t
-
struct Index2Layer : public faiss::IndexFlatCodes
- #include <Index2Layer.h>
Same as an IndexIVFPQ without the inverted lists: codes are stored sequentially
The class is mainly inteded to store encoded vectors that can be accessed randomly, the search function is not implemented.
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
Index2Layer(Index *quantizer, size_t nlist, int M, int nbit = 8, MetricType metric = METRIC_L2)
-
Index2Layer()
-
~Index2Layer()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
not implemented
-
virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void transfer_to_IVFPQ(IndexIVFPQ &other) const
transfer the flat codes to an IVFPQ index
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
reconstruction using the codec interface
-
virtual void reconstruct(idx_t key, float *recons) const override¶
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override¶
remove some ids. NB that Because of the structure of the indexing structure, the semantics of this operation are different from the usual ones: the new ids are shifted
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const¶
a FlatCodesDistanceComputer offers a distance_to_code method
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
Level1Quantizer q1
first level quantizer
-
ProductQuantizer pq
second level quantizer is always a PQ
-
size_t code_size_1
size of the code for the first level (ceil(log8(q1.nlist)))
-
size_t code_size_2
size of the code for the second level
-
size_t code_size¶
-
std::vector<uint8_t> codes¶
encoded dataset, size ntotal * code_size
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
struct IndexAdditiveQuantizer : public faiss::IndexFlatCodes
- #include <IndexAdditiveQuantizer.h>
Abstract class for additive quantizers. The search functions are in common.
Subclassed by faiss::IndexLocalSearchQuantizer, faiss::IndexResidualQuantizer
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
explicit IndexAdditiveQuantizer(idx_t d = 0, AdditiveQuantizer *aq = nullptr, MetricType metric = METRIC_L2)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
reconstruction using the codec interface
-
virtual void reconstruct(idx_t key, float *recons) const override¶
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override¶
remove some ids. NB that Because of the structure of the indexing structure, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void train(idx_t n, const float *x)¶
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexAdditiveQuantizerFastScan : public faiss::IndexFastScan
- #include <IndexAdditiveQuantizerFastScan.h>
Fast scan version of IndexAQ. Works for 4-bit AQ for now.
The codes are not stored sequentially but grouped in blocks of size bbs. This makes it possible to compute distances quickly with SIMD instructions.
Implementations: 12: blocked loop with internal loop on Q with qbs 13: same with reservoir accumulator to store results 14: no qbs with heap accumulator 15: no qbs with reservoir accumulator
Subclassed by faiss::IndexLocalSearchQuantizerFastScan, faiss::IndexResidualQuantizerFastScan
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
explicit IndexAdditiveQuantizerFastScan(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)
-
void init(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)
-
IndexAdditiveQuantizerFastScan()
-
~IndexAdditiveQuantizerFastScan() override
-
explicit IndexAdditiveQuantizerFastScan(const IndexAdditiveQuantizer &orig, int bbs = 32)
build from an existing IndexAQ
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override
-
virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
Decode a set of vectors.
NOTE: The codes in the IndexAdditiveQuantizerFastScan object are non- contiguous. But this method requires a contiguous representation.
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * code_size
x – output vectors, size n * d
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)¶
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void add(idx_t n, const float *x) override¶
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
template<bool is_max, class Scaler>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class Cfloat, class Scaler>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const Scaler &scaler) const¶
-
virtual void reconstruct(idx_t key, float *recons) const override¶
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
Public Members
-
AdditiveQuantizer *aq
-
bool rescale_norm = true
-
int norm_scale = 1
-
size_t max_train_points = 0
-
int implem = 0¶
-
int skip = 0¶
-
int bbs¶
-
int qbs = 0¶
-
size_t M¶
-
size_t nbits¶
-
size_t ksub¶
-
size_t code_size¶
-
size_t ntotal2¶
-
size_t M2¶
-
AlignedTable<uint8_t> codes¶
-
const uint8_t *orig_codes = nullptr¶
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexBinary
- #include <IndexBinary.h>
Abstract structure for a binary index.
Supports adding vertices and searching them.
All queries are symmetric because there is no distinction between codes and vectors.
Subclassed by faiss::gpu::GpuIndexBinaryFlat, faiss::IndexBinaryFlat, faiss::IndexBinaryFromFloat, faiss::IndexBinaryHash, faiss::IndexBinaryHNSW, faiss::IndexBinaryIVF, faiss::IndexBinaryMultiHash
Public Types
-
using component_t = uint8_t
-
using distance_t = int32_t
Public Functions
-
inline explicit IndexBinary(idx_t d = 0, MetricType metric = METRIC_L2)
-
virtual ~IndexBinary()
-
virtual void train(idx_t n, const uint8_t *x)
Perform training on a representative set of vectors.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d / 8
-
virtual void add(idx_t n, const uint8_t *x) = 0
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels) const = 0
Query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result) const
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual void reset() = 0
Removes all elements from the database.
-
virtual size_t remove_ids(const IDSelector &sel)
Removes IDs from the index. Not supported by all indexes.
-
virtual void reconstruct(idx_t key, uint8_t *recons) const
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void display() const
Display the actual class name and some more info.
Public Members
-
int d
vector dimension
-
int code_size
number of bytes per vector ( = d / 8 )
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
using component_t = uint8_t
-
struct IndexBinaryFlat : public faiss::IndexBinary
- #include <IndexBinaryFlat.h>
Index that stores the full vectors and performs exhaustive search.
Public Functions
-
explicit IndexBinaryFlat(idx_t d)
-
virtual void add(idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void reset() override
Removes all elements from the database.
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels) const override
Query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result) const override
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, uint8_t *recons) const override
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual size_t remove_ids(const IDSelector &sel) override
Remove some ids. Note that because of the indexing structure, the semantics of this operation are different from the usual ones: the new ids are shifted.
-
inline IndexBinaryFlat()
-
virtual void train(idx_t n, const uint8_t *x)¶
Perform training on a representative set of vectors.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const¶
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void display() const¶
Display the actual class name and some more info.
Public Members
-
std::vector<uint8_t> xb
database vectors, size ntotal * d / 8
-
bool use_heap = true
Select between using a heap or counting to select the k smallest values when scanning inverted lists.
-
size_t query_batch_size = 32
-
int d¶
vector dimension
-
int code_size¶
number of bytes per vector ( = d / 8 )
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
explicit IndexBinaryFlat(idx_t d)
-
struct IndexBinaryFromFloat : public faiss::IndexBinary
- #include <IndexBinaryFromFloat.h>
IndexBinary backed by a float Index.
Supports adding vertices and searching them.
All queries are symmetric because there is no distinction between codes and vectors.
Public Functions
-
IndexBinaryFromFloat()
-
explicit IndexBinaryFromFloat(Index *index)
-
~IndexBinaryFromFloat()
-
virtual void add(idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void reset() override
Removes all elements from the database.
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels) const override
Query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void train(idx_t n, const uint8_t *x) override
Perform training on a representative set of vectors.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result) const¶
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const¶
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
Removes IDs from the index. Not supported by all indexes.
-
virtual void reconstruct(idx_t key, uint8_t *recons) const¶
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void display() const¶
Display the actual class name and some more info.
-
IndexBinaryFromFloat()
-
struct IndexBinaryHash : public faiss::IndexBinary
- #include <IndexBinaryHash.h>
just uses the b first bits as a hash value
Public Types
-
using InvertedListMap = std::unordered_map<idx_t, InvertedList>
-
using component_t = uint8_t¶
-
using distance_t = int32_t¶
Public Functions
-
IndexBinaryHash(int d, int b)
-
IndexBinaryHash()
-
virtual void reset() override
Removes all elements from the database.
-
virtual void add(idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids) override
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result) const override
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels) const override
Query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
void display() const
-
size_t hashtable_size() const
-
virtual void train(idx_t n, const uint8_t *x)¶
Perform training on a representative set of vectors.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d / 8
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const¶
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
Removes IDs from the index. Not supported by all indexes.
-
virtual void reconstruct(idx_t key, uint8_t *recons) const¶
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
Public Members
-
InvertedListMap invlists
-
int b
-
int nflip
-
int d¶
vector dimension
-
int code_size¶
number of bytes per vector ( = d / 8 )
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
struct InvertedList
Public Functions
-
void add(idx_t id, size_t code_size, const uint8_t *code)
-
void add(idx_t id, size_t code_size, const uint8_t *code)
-
using InvertedListMap = std::unordered_map<idx_t, InvertedList>
-
struct IndexBinaryHashStats
Public Functions
-
inline IndexBinaryHashStats()
-
void reset()
Public Members
-
size_t nq
-
size_t n0
-
size_t nlist
-
size_t ndis
-
inline IndexBinaryHashStats()
-
struct IndexBinaryHNSW : public faiss::IndexBinary
- #include <IndexBinaryHNSW.h>
The HNSW index is a normal random-access index with a HNSW link structure built on top
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using component_t = uint8_t¶
-
using distance_t = int32_t¶
Public Functions
-
explicit IndexBinaryHNSW()
-
explicit IndexBinaryHNSW(int d, int M = 32)
-
explicit IndexBinaryHNSW(IndexBinary *storage, int M = 32)
-
~IndexBinaryHNSW() override
-
DistanceComputer *get_distance_computer() const
-
virtual void add(idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void train(idx_t n, const uint8_t *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels) const override
entry point for search
-
virtual void reconstruct(idx_t key, uint8_t *recons) const override
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void reset() override
Removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result) const¶
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const¶
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
Removes IDs from the index. Not supported by all indexes.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void display() const¶
Display the actual class name and some more info.
Public Members
-
HNSW hnsw
-
bool own_fields
-
IndexBinary *storage
-
int d¶
vector dimension
-
int code_size¶
number of bytes per vector ( = d / 8 )
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
typedef HNSW::storage_idx_t storage_idx_t
-
struct IndexBinaryIVF : public faiss::IndexBinary
- #include <IndexBinaryIVF.h>
Index based on a inverted file (IVF)
In the inverted file, the quantizer (an IndexBinary instance) provides a quantization index for each vector to be added. The quantization index maps to a list (aka inverted list or posting list), where the id of the vector is stored.
Otherwise the object is similar to the IndexIVF
Public Functions
-
IndexBinaryIVF(IndexBinary *quantizer, size_t d, size_t nlist)
The Inverted file takes a quantizer (an IndexBinary) on input, which implements the function mapping a vector to a list identifier. The pointer is borrowed: the quantizer should not be deleted while the IndexBinaryIVF is in use.
-
IndexBinaryIVF()
-
~IndexBinaryIVF() override
-
virtual void reset() override
Removes all elements from the database.
-
virtual void train(idx_t n, const uint8_t *x) override
Trains the quantizer.
-
virtual void add(idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids) override
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
void add_core(idx_t n, const uint8_t *x, const idx_t *xids, const idx_t *precomputed_idx)
Implementation of vector addition where the vector assignments are predefined.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void search_preassigned(idx_t n, const uint8_t *x, idx_t k, const idx_t *assign, const int32_t *centroid_dis, int32_t *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr) const
Search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. search() calls this.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
-
virtual BinaryInvertedListScanner *get_InvertedListScanner(bool store_pairs = false) const
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result) const override
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
void range_search_preassigned(idx_t n, const uint8_t *x, int radius, const idx_t *assign, const int32_t *centroid_dis, RangeSearchResult *result) const
-
virtual void reconstruct(idx_t key, uint8_t *recons) const override
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d / 8
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d / 8)
-
virtual void reconstruct_from_offset(idx_t list_no, idx_t offset, uint8_t *recons) const
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void merge_from(IndexBinaryIVF &other, idx_t add_id)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal
-
inline size_t get_list_size(size_t list_no) const
-
void make_direct_map(bool new_maintain_direct_map = true)
intialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const¶
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
void display() const¶
Display the actual class name and some more info.
Public Members
-
InvertedLists *invlists
Access to the actual data.
-
bool own_invlists
-
size_t nprobe
number of probes at query time
-
size_t max_codes
max nb of codes to visit to do a query
-
bool use_heap = true
Select between using a heap or counting to select the k smallest values when scanning inverted lists.
-
DirectMap direct_map
map for direct access to the elements. Enables reconstruct().
-
IndexBinary *quantizer
quantizer that maps vectors to inverted lists
-
size_t nlist
number of possible key values
-
bool own_fields
whether object owns the quantizer
-
ClusteringParameters cp
to override default clustering params
-
Index *clustering_index
to override index used during clustering
-
int d¶
vector dimension
-
int code_size¶
number of bytes per vector ( = d / 8 )
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
IndexBinaryIVF(IndexBinary *quantizer, size_t d, size_t nlist)
-
struct IndexBinaryMultiHash : public faiss::IndexBinary
- #include <IndexBinaryHash.h>
just uses the b first bits as a hash value
Public Functions
-
IndexBinaryMultiHash(int d, int nhash, int b)
-
IndexBinaryMultiHash()
-
~IndexBinaryMultiHash()
-
virtual void reset() override
Removes all elements from the database.
-
virtual void add(idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result) const override
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
virtual void search(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels) const override
Query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
size_t hashtable_size() const
-
virtual void train(idx_t n, const uint8_t *x)¶
Perform training on a representative set of vectors.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const¶
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
Removes IDs from the index. Not supported by all indexes.
-
virtual void reconstruct(idx_t key, uint8_t *recons) const¶
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void display() const¶
Display the actual class name and some more info.
Public Members
-
IndexBinaryFlat *storage
-
bool own_fields
-
std::vector<Map> maps
-
int nhash
nb of hash maps
-
int b
nb bits per hash map
-
int nflip
nb bit flips to use at search time
-
int d¶
vector dimension
-
int code_size¶
number of bytes per vector ( = d / 8 )
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
IndexBinaryMultiHash(int d, int nhash, int b)
-
struct IndexFastScan : public faiss::Index
- #include <IndexFastScan.h>
Fast scan version of IndexPQ and IndexAQ. Works for 4-bit PQ and AQ for now.
The codes are not stored sequentially but grouped in blocks of size bbs. This makes it possible to compute distances quickly with SIMD instructions.
Implementations: 12: blocked loop with internal loop on Q with qbs 13: same with reservoir accumulator to store results 14: no qbs with heap accumulator 15: no qbs with reservoir accumulator
Subclassed by faiss::IndexAdditiveQuantizerFastScan, faiss::IndexPQFastScan
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)
-
IndexFastScan()
-
virtual void reset() override
removes all elements from the database.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const = 0
-
virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const = 0
-
void compute_quantized_LUT(idx_t n, const float *x, uint8_t *lut, float *normalizers) const
-
template<bool is_max, class Scaler>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const
-
template<class Cfloat, class Scaler>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const
-
template<class C, class Scaler>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const Scaler &scaler) const
-
template<class C, class Scaler>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const Scaler &scaler) const
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void train(idx_t n, const float *x)¶
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
int implem = 0
-
int skip = 0
-
int bbs
-
int qbs = 0
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t code_size
-
size_t ntotal2
-
size_t M2
-
AlignedTable<uint8_t> codes
-
const uint8_t *orig_codes = nullptr
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
struct IndexFlat : public faiss::IndexFlatCodes
- #include <IndexFlat.h>
Index that stores the full vectors and performs exhaustive search
Subclassed by faiss::IndexFlatIP, faiss::IndexFlatL2
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
explicit IndexFlat(idx_t d, MetricType metric = METRIC_L2)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
void compute_distance_subset(idx_t n, const float *x, idx_t k, float *distances, const idx_t *labels) const
compute distance with a subset of vectors
- Parameters:
x – query vectors, size n * d
labels – indices of the vectors that should be compared for each query vector, size n * k
distances – corresponding output distances, size n * k
-
inline float *get_xb()
-
inline const float *get_xb() const
-
inline IndexFlat()
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
reconstruction using the codec interface
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override¶
remove some ids. NB that Because of the structure of the indexing structure, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void train(idx_t n, const float *x)¶
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
using idx_t = int64_t¶
-
struct IndexFlat1D : public faiss::IndexFlatL2
- #include <IndexFlat.h>
optimized version for 1D “vectors”.
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
explicit IndexFlat1D(bool continuous_update = true)
-
void update_permutation()
if not continuous_update, call this between the last add and the first search
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
Warn: the distances returned are L1 not L2.
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
void compute_distance_subset(idx_t n, const float *x, idx_t k, float *distances, const idx_t *labels) const
compute distance with a subset of vectors
- Parameters:
x – query vectors, size n * d
labels – indices of the vectors that should be compared for each query vector, size n * k
distances – corresponding output distances, size n * k
-
inline float *get_xb()
-
inline const float *get_xb() const
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
reconstruction using the codec interface
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override¶
remove some ids. NB that Because of the structure of the indexing structure, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void train(idx_t n, const float *x)¶
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
bool continuous_update
is the permutation updated continuously?
-
std::vector<idx_t> perm
sorted database indices
-
size_t code_size¶
-
std::vector<uint8_t> codes¶
encoded dataset, size ntotal * code_size
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
struct IndexFlatCodes : public faiss::Index
- #include <IndexFlatCodes.h>
Index that encodes all vectors as fixed-size codes (size code_size). Storage is in the codes vector
Subclassed by faiss::Index2Layer, faiss::IndexAdditiveQuantizer, faiss::IndexFlat, faiss::IndexLSH, faiss::IndexPQ, faiss::IndexScalarQuantizer
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexFlatCodes()
-
IndexFlatCodes(size_t code_size, idx_t d, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
default add uses sa_encode
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
reconstruction using the codec interface
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override
remove some ids. NB that Because of the structure of the indexing structure, the semantics of this operation are different from the usual ones: the new ids are shifted
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const
a FlatCodesDistanceComputer offers a distance_to_code method
-
inline virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void train(idx_t n, const float *x)¶
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const = 0¶
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
size_t code_size
-
std::vector<uint8_t> codes
encoded dataset, size ntotal * code_size
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
struct IndexFlatIP : public faiss::IndexFlat
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
inline explicit IndexFlatIP(idx_t d)
-
inline IndexFlatIP()
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
void compute_distance_subset(idx_t n, const float *x, idx_t k, float *distances, const idx_t *labels) const
compute distance with a subset of vectors
- Parameters:
x – query vectors, size n * d
labels – indices of the vectors that should be compared for each query vector, size n * k
distances – corresponding output distances, size n * k
-
inline float *get_xb()
-
inline const float *get_xb() const
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
reconstruction using the codec interface
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override¶
remove some ids. NB that Because of the structure of the indexing structure, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void train(idx_t n, const float *x)¶
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
using idx_t = int64_t¶
-
struct IndexFlatL2 : public faiss::IndexFlat
Subclassed by faiss::IndexFlat1D
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
inline explicit IndexFlatL2(idx_t d)
-
inline IndexFlatL2()
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
void compute_distance_subset(idx_t n, const float *x, idx_t k, float *distances, const idx_t *labels) const
compute distance with a subset of vectors
- Parameters:
x – query vectors, size n * d
labels – indices of the vectors that should be compared for each query vector, size n * k
distances – corresponding output distances, size n * k
-
inline float *get_xb()
-
inline const float *get_xb() const
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
reconstruction using the codec interface
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override¶
remove some ids. NB that Because of the structure of the indexing structure, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void train(idx_t n, const float *x)¶
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
using idx_t = int64_t¶
-
struct IndexHNSW : public faiss::Index
- #include <IndexHNSW.h>
The HNSW index is a normal random-access index with a HNSW link structure built on top
Subclassed by faiss::IndexHNSW2Level, faiss::IndexHNSWFlat, faiss::IndexHNSWPQ, faiss::IndexHNSWSQ
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
explicit IndexHNSW(int d = 0, int M = 32, MetricType metric = METRIC_L2)
-
explicit IndexHNSW(Index *storage, int M = 32)
-
~IndexHNSW() override
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
entry point for search
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
void shrink_level_0_neighbors(int size)
-
void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1) const
Perform search only on level 0, given the starting points for each vertex.
- Parameters:
search_type – 1:perform one search per nprobe, 2: enqueue all entry points
-
void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)
alternative graph building
-
void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)
alternative graph building
-
void reorder_links()
-
void link_singletons()
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
HNSW hnsw
-
bool own_fields
-
Index *storage
-
ReconstructFromNeighbors *reconstruct_from_neighbors
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
typedef HNSW::storage_idx_t storage_idx_t
-
struct IndexHNSW2Level : public faiss::IndexHNSW
- #include <IndexHNSW.h>
2-level code structure with fast random access
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexHNSW2Level()
-
IndexHNSW2Level(Index *quantizer, size_t nlist, int m_pq, int M)
-
void flip_to_ivf()
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
entry point for search
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
void shrink_level_0_neighbors(int size)
-
void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1) const
Perform search only on level 0, given the starting points for each vertex.
- Parameters:
search_type – 1:perform one search per nprobe, 2: enqueue all entry points
-
void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)
alternative graph building
-
void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)
alternative graph building
-
void reorder_links()
-
void link_singletons()
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
HNSW hnsw
-
bool own_fields
-
Index *storage
-
ReconstructFromNeighbors *reconstruct_from_neighbors
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
typedef HNSW::storage_idx_t storage_idx_t
-
struct IndexHNSWFlat : public faiss::IndexHNSW
- #include <IndexHNSW.h>
Flat index topped with with a HNSW structure to access elements more efficiently.
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexHNSWFlat()
-
IndexHNSWFlat(int d, int M, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
entry point for search
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
void shrink_level_0_neighbors(int size)
-
void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1) const
Perform search only on level 0, given the starting points for each vertex.
- Parameters:
search_type – 1:perform one search per nprobe, 2: enqueue all entry points
-
void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)
alternative graph building
-
void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)
alternative graph building
-
void reorder_links()
-
void link_singletons()
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
HNSW hnsw
-
bool own_fields
-
Index *storage
-
ReconstructFromNeighbors *reconstruct_from_neighbors
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
typedef HNSW::storage_idx_t storage_idx_t
-
struct IndexHNSWPQ : public faiss::IndexHNSW
- #include <IndexHNSW.h>
PQ index topped with with a HNSW structure to access elements more efficiently.
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexHNSWPQ()
-
IndexHNSWPQ(int d, int pq_m, int M)
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
entry point for search
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
void shrink_level_0_neighbors(int size)
-
void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1) const
Perform search only on level 0, given the starting points for each vertex.
- Parameters:
search_type – 1:perform one search per nprobe, 2: enqueue all entry points
-
void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)
alternative graph building
-
void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)
alternative graph building
-
void reorder_links()
-
void link_singletons()
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
HNSW hnsw
-
bool own_fields
-
Index *storage
-
ReconstructFromNeighbors *reconstruct_from_neighbors
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
typedef HNSW::storage_idx_t storage_idx_t
-
struct IndexHNSWSQ : public faiss::IndexHNSW
- #include <IndexHNSW.h>
SQ index topped with with a HNSW structure to access elements more efficiently.
Public Types
-
typedef HNSW::storage_idx_t storage_idx_t
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexHNSWSQ()
-
IndexHNSWSQ(int d, ScalarQuantizer::QuantizerType qtype, int M, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
entry point for search
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
void shrink_level_0_neighbors(int size)
-
void search_level_0(idx_t n, const float *x, idx_t k, const storage_idx_t *nearest, const float *nearest_d, float *distances, idx_t *labels, int nprobe = 1, int search_type = 1) const
Perform search only on level 0, given the starting points for each vertex.
- Parameters:
search_type – 1:perform one search per nprobe, 2: enqueue all entry points
-
void init_level_0_from_knngraph(int k, const float *D, const idx_t *I)
alternative graph building
-
void init_level_0_from_entry_points(int npt, const storage_idx_t *points, const storage_idx_t *nearests)
alternative graph building
-
void reorder_links()
-
void link_singletons()
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
HNSW hnsw
-
bool own_fields
-
Index *storage
-
ReconstructFromNeighbors *reconstruct_from_neighbors
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
typedef HNSW::storage_idx_t storage_idx_t
-
template<typename IndexT>
struct IndexIDMap2Template : public faiss::IndexIDMapTemplate<IndexT> - #include <MetaIndexes.h>
same as IndexIDMap but also provides an efficient reconstruction implementation via a 2-way index
Public Types
-
using idx_t = typename IndexT::idx_t
-
using component_t = typename IndexT::component_t
-
using distance_t = typename IndexT::distance_t
Public Functions
-
explicit IndexIDMap2Template(IndexT *index)
-
void construct_rev_map()
make the rev_map from scratch
-
void add_with_ids(idx_t n, const component_t *x, const idx_t *xids) override
-
size_t remove_ids(const IDSelector &sel) override
-
void reconstruct(idx_t key, component_t *recons) const override
-
inline ~IndexIDMap2Template() override
-
inline IndexIDMap2Template()
-
void add(idx_t n, const component_t *x) override
this will fail. Use add_with_ids
-
void search(idx_t n, const component_t *x, idx_t k, distance_t *distances, idx_t *labels) const override
-
void train(idx_t n, const component_t *x) override
-
void reset() override
-
void range_search(idx_t n, const component_t *x, distance_t radius, RangeSearchResult *result) const override
-
using idx_t = typename IndexT::idx_t
-
template<typename IndexT>
struct IndexIDMapTemplate : public IndexT - #include <MetaIndexes.h>
Index that translates search results to ids
Subclassed by faiss::IndexIDMap2Template< IndexT >
Public Types
-
using idx_t = typename IndexT::idx_t
-
using component_t = typename IndexT::component_t
-
using distance_t = typename IndexT::distance_t
Public Functions
-
explicit IndexIDMapTemplate(IndexT *index)
-
void add_with_ids(idx_t n, const component_t *x, const idx_t *xids) override
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
void add(idx_t n, const component_t *x) override
this will fail. Use add_with_ids
-
void search(idx_t n, const component_t *x, idx_t k, distance_t *distances, idx_t *labels) const override
-
void train(idx_t n, const component_t *x) override
-
void reset() override
-
size_t remove_ids(const IDSelector &sel) override
remove ids adapted to IndexFlat
-
void range_search(idx_t n, const component_t *x, distance_t radius, RangeSearchResult *result) const override
-
~IndexIDMapTemplate() override
-
inline IndexIDMapTemplate()
-
using idx_t = typename IndexT::idx_t
-
struct IndexIVF : public faiss::Index, public faiss::Level1Quantizer
- #include <IndexIVF.h>
Index based on a inverted file (IVF)
In the inverted file, the quantizer (an Index instance) provides a quantization index for each vector to be added. The quantization index maps to a list (aka inverted list or posting list), where the id of the vector is stored.
The inverted list object is required only after trainng. If none is set externally, an ArrayInvertedLists is used automatically.
At search time, the vector to be searched is also quantized, and only the list corresponding to the quantization index is searched. This speeds up the search by making it non-exhaustive. This can be relaxed using multi-probe search: a few (nprobe) quantization indices are selected and several inverted lists are visited.
Sub-classes implement a post-filtering of the index that refines the distance estimation from the query to databse vectors.
Subclassed by faiss::IndexIVFAdditiveQuantizer, faiss::IndexIVFFastScan, faiss::IndexIVFFlat, faiss::IndexIVFPQ, faiss::IndexIVFScalarQuantizer, faiss::IndexIVFSpectralHash
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexIVF(Index *quantizer, size_t d, size_t nlist, size_t code_size, MetricType metric = METRIC_L2)
The Inverted file takes a quantizer (an Index) on input, which implements the function mapping a vector to a list identifier.
-
virtual void reset() override
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override
Trains the quantizer and calls train_residual to train sub-quantizers.
-
virtual void add(idx_t n, const float *x) override
Calls add_with_ids with NULL ids.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx)
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listno = false) const = 0
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void train_residual(idx_t n, const float *x)
Sub-classes that encode the residuals can train their encoders here does nothing by default
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs = false) const
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct(idx_t key, float *recons) const override
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
void check_compatible_for_merge(const IndexIVF &other) const
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(IndexIVF &other, idx_t add_id)
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal
-
virtual void copy_subset_to(IndexIVF &other, int subset_type, idx_t a1, idx_t a2) const
copy a subset of the entries index to the other index
if subset_type == 0: copies ids in [a1, a2) if subset_type == 1: copies ids if id % a1 == a2 if subset_type == 2: copies inverted lists such that a1 elements are left before and a2 elements are after
-
~IndexIVF() override
-
inline size_t get_list_size(size_t list_no) const
-
void make_direct_map(bool new_maintain_direct_map = true)
intialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
IndexIVF()
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
Public Members
-
InvertedLists *invlists
Access to the actual data.
-
bool own_invlists
-
size_t code_size
code size per vector in bytes
-
size_t nprobe
number of probes at query time
-
size_t max_codes
max nb of codes to visit to do a query
-
int parallel_mode
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024
-
DirectMap direct_map
optional map that maps back ids to invlist entries. This enables reconstruct()
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
Index *quantizer
quantizer that maps vectors to inverted lists
-
size_t nlist
number of possible key values
-
char quantizer_trains_alone
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields
whether object owns the quantizer (false by default)
-
ClusteringParameters cp
to override default clustering params
-
Index *clustering_index
to override index used during clustering
-
using idx_t = int64_t¶
-
struct IndexIVFAdditiveQuantizer : public faiss::IndexIVF
- #include <IndexIVFAdditiveQuantizer.h>
Abstract class for IVF additive quantizers. The search functions are in common.
Subclassed by faiss::IndexIVFLocalSearchQuantizer, faiss::IndexIVFResidualQuantizer
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexIVFAdditiveQuantizer(AdditiveQuantizer *aq, Index *quantizer, size_t d, size_t nlist, MetricType metric = METRIC_L2)
-
explicit IndexIVFAdditiveQuantizer(AdditiveQuantizer *aq)
-
virtual void train_residual(idx_t n, const float *x) override
Sub-classes that encode the residuals can train their encoders here does nothing by default
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void sa_decode(idx_t n, const uint8_t *codes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
~IndexIVFAdditiveQuantizer() override
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override¶
default implementation that calls encode_vectors
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx)¶
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)¶
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override¶
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
-
virtual void reconstruct(idx_t key, float *recons) const override¶
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)¶
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const override¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const¶
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual size_t remove_ids(const IDSelector &sel) override¶
Dataset manipulation functions.
-
void check_compatible_for_merge(const IndexIVF &other) const¶
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(IndexIVF &other, idx_t add_id)¶
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal
-
virtual void copy_subset_to(IndexIVF &other, int subset_type, idx_t a1, idx_t a2) const¶
copy a subset of the entries index to the other index
if subset_type == 0: copies ids in [a1, a2) if subset_type == 1: copies ids if id % a1 == a2 if subset_type == 2: copies inverted lists such that a1 elements are left before and a2 elements are after
-
inline size_t get_list_size(size_t list_no) const¶
-
void make_direct_map(bool new_maintain_direct_map = true)¶
intialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)¶
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const¶
compute the number of bytes required to store list ids
Public Members
-
AdditiveQuantizer *aq
-
bool by_residual = true
-
int use_precomputed_table = 0
-
InvertedLists *invlists¶
Access to the actual data.
-
bool own_invlists¶
-
size_t code_size¶
code size per vector in bytes
-
size_t nprobe¶
number of probes at query time
-
size_t max_codes¶
max nb of codes to visit to do a query
-
int parallel_mode¶
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024¶
-
DirectMap direct_map¶
optional map that maps back ids to invlist entries. This enables reconstruct()
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
size_t nlist¶
number of possible key values
-
char quantizer_trains_alone¶
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields¶
whether object owns the quantizer (false by default)
-
ClusteringParameters cp¶
to override default clustering params
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexIVFAdditiveQuantizerFastScan : public faiss::IndexIVFFastScan
- #include <IndexIVFAdditiveQuantizerFastScan.h>
Fast scan version of IVFAQ. Works for 4-bit AQ for now.
The codes in the inverted lists are not stored sequentially but grouped in blocks of size bbs. This makes it possible to very quickly compute distances with SIMD instructions.
Implementations (implem): 0: auto-select implementation (default) 1: orig’s search, re-implemented 2: orig’s search, re-ordered by invlist 10: optimizer int16 search, collect results in heap, no qbs 11: idem, collect results in reservoir 12: optimizer int16 search, collect results in heap, uses qbs 13: idem, collect results in reservoir
Subclassed by faiss::IndexIVFLocalSearchQuantizerFastScan, faiss::IndexIVFResidualQuantizerFastScan
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexIVFAdditiveQuantizerFastScan(Index *quantizer, AdditiveQuantizer *aq, size_t d, size_t nlist, MetricType metric = METRIC_L2, int bbs = 32)
-
void init(AdditiveQuantizer *aq, size_t nlist, MetricType metric, int bbs)
-
IndexIVFAdditiveQuantizerFastScan()
-
~IndexIVFAdditiveQuantizerFastScan() override
-
explicit IndexIVFAdditiveQuantizerFastScan(const IndexIVFAdditiveQuantizer &orig, int bbs = 32)
-
virtual void train_residual(idx_t n, const float *x) override
Sub-classes that encode the residuals can train their encoders here does nothing by default
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listno = false) const override
same as the regular IVFAQ encoder. The codes are not reorganized by blocks a that point
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
assign the vectors, then call search_preassign
-
virtual bool lookup_table_is_3d() const override
-
virtual void compute_LUT(size_t n, const float *x, const idx_t *coarse_ids, const float *coarse_dis, AlignedTable<float> &dis_tables, AlignedTable<float> &biases) const override
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void init_fastscan(size_t M, size_t nbits, size_t nlist, MetricType metric, int bbs)¶
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override¶
default implementation that calls encode_vectors
-
void compute_LUT_uint8(size_t n, const float *x, const idx_t *coarse_ids, const float *coarse_dis, AlignedTable<uint8_t> &dis_tables, AlignedTable<uint16_t> &biases, float *normalizers) const¶
-
template<bool is_max, class Scaler>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_1(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_2(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_10(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, size_t *ndis_out, size_t *nlist_out, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, size_t *ndis_out, size_t *nlist_out, const Scaler &scaler) const¶
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override¶
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
void reconstruct_orig_invlists()¶
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx)¶
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)¶
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs = false) const¶
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct(idx_t key, float *recons) const override¶
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)¶
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const override¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual size_t remove_ids(const IDSelector &sel) override¶
Dataset manipulation functions.
-
void check_compatible_for_merge(const IndexIVF &other) const¶
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(IndexIVF &other, idx_t add_id)¶
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal
-
virtual void copy_subset_to(IndexIVF &other, int subset_type, idx_t a1, idx_t a2) const¶
copy a subset of the entries index to the other index
if subset_type == 0: copies ids in [a1, a2) if subset_type == 1: copies ids if id % a1 == a2 if subset_type == 2: copies inverted lists such that a1 elements are left before and a2 elements are after
-
inline size_t get_list_size(size_t list_no) const¶
-
void make_direct_map(bool new_maintain_direct_map = true)¶
intialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)¶
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const¶
compute the number of bytes required to store list ids
Public Members
-
AdditiveQuantizer *aq
-
bool rescale_norm = false
-
int norm_scale = 1
-
size_t max_train_points
-
int bbs¶
-
size_t M¶
-
size_t nbits¶
-
size_t ksub¶
-
size_t M2¶
-
int implem = 0¶
-
int skip = 0¶
-
bool by_residual = false¶
-
int qbs = 0¶
-
size_t qbs2 = 0¶
-
InvertedLists *orig_invlists = nullptr¶
orig’s inverted lists (for debugging)
-
InvertedLists *invlists¶
Access to the actual data.
-
bool own_invlists¶
-
size_t code_size¶
code size per vector in bytes
-
size_t nprobe¶
number of probes at query time
-
size_t max_codes¶
max nb of codes to visit to do a query
-
int parallel_mode¶
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024¶
-
DirectMap direct_map¶
optional map that maps back ids to invlist entries. This enables reconstruct()
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
size_t nlist¶
number of possible key values
-
char quantizer_trains_alone¶
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields¶
whether object owns the quantizer (false by default)
-
ClusteringParameters cp¶
to override default clustering params
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexIVFFastScan : public faiss::IndexIVF
- #include <IndexIVFFastScan.h>
Fast scan version of IVFPQ and IVFAQ. Works for 4-bit PQ/AQ for now.
The codes in the inverted lists are not stored sequentially but grouped in blocks of size bbs. This makes it possible to very quickly compute distances with SIMD instructions.
Implementations (implem): 0: auto-select implementation (default) 1: orig’s search, re-implemented 2: orig’s search, re-ordered by invlist 10: optimizer int16 search, collect results in heap, no qbs 11: idem, collect results in reservoir 12: optimizer int16 search, collect results in heap, uses qbs 13: idem, collect results in reservoir
Subclassed by faiss::IndexIVFAdditiveQuantizerFastScan, faiss::IndexIVFPQFastScan
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexIVFFastScan(Index *quantizer, size_t d, size_t nlist, size_t code_size, MetricType metric = METRIC_L2)
-
IndexIVFFastScan()
-
void init_fastscan(size_t M, size_t nbits, size_t nlist, MetricType metric, int bbs)
-
~IndexIVFFastScan() override
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
virtual bool lookup_table_is_3d() const = 0
-
virtual void compute_LUT(size_t n, const float *x, const idx_t *coarse_ids, const float *coarse_dis, AlignedTable<float> &dis_tables, AlignedTable<float> &biases) const = 0
-
void compute_LUT_uint8(size_t n, const float *x, const idx_t *coarse_ids, const float *coarse_dis, AlignedTable<uint8_t> &dis_tables, AlignedTable<uint16_t> &biases, float *normalizers) const
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
assign the vectors, then call search_preassign
-
template<bool is_max, class Scaler>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const
-
template<class C, class Scaler>
void search_implem_1(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const
-
template<class C, class Scaler>
void search_implem_2(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const
-
template<class C, class Scaler>
void search_implem_10(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, size_t *ndis_out, size_t *nlist_out, const Scaler &scaler) const
-
template<class C, class Scaler>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, size_t *ndis_out, size_t *nlist_out, const Scaler &scaler) const
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
void reconstruct_orig_invlists()
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx)¶
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listno = false) const = 0¶
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)¶
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void train_residual(idx_t n, const float *x)¶
Sub-classes that encode the residuals can train their encoders here does nothing by default
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs = false) const¶
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct(idx_t key, float *recons) const override¶
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)¶
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const override¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual size_t remove_ids(const IDSelector &sel) override¶
Dataset manipulation functions.
-
void check_compatible_for_merge(const IndexIVF &other) const¶
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(IndexIVF &other, idx_t add_id)¶
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal
-
virtual void copy_subset_to(IndexIVF &other, int subset_type, idx_t a1, idx_t a2) const¶
copy a subset of the entries index to the other index
if subset_type == 0: copies ids in [a1, a2) if subset_type == 1: copies ids if id % a1 == a2 if subset_type == 2: copies inverted lists such that a1 elements are left before and a2 elements are after
-
inline size_t get_list_size(size_t list_no) const¶
-
void make_direct_map(bool new_maintain_direct_map = true)¶
intialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)¶
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const¶
compute the number of bytes required to store list ids
Public Members
-
int bbs
-
size_t M
-
size_t nbits
-
size_t ksub
-
size_t M2
-
int implem = 0
-
int skip = 0
-
bool by_residual = false
-
int qbs = 0
-
size_t qbs2 = 0
-
InvertedLists *orig_invlists = nullptr
orig’s inverted lists (for debugging)
-
InvertedLists *invlists¶
Access to the actual data.
-
bool own_invlists¶
-
size_t code_size¶
code size per vector in bytes
-
size_t nprobe¶
number of probes at query time
-
size_t max_codes¶
max nb of codes to visit to do a query
-
int parallel_mode¶
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024¶
-
DirectMap direct_map¶
optional map that maps back ids to invlist entries. This enables reconstruct()
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
size_t nlist¶
number of possible key values
-
char quantizer_trains_alone¶
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields¶
whether object owns the quantizer (false by default)
-
ClusteringParameters cp¶
to override default clustering params
-
using idx_t = int64_t¶
-
struct IndexIVFFlat : public faiss::IndexIVF
- #include <IndexIVFFlat.h>
Inverted file with stored vectors. Here the inverted file pre-selects the vectors to be searched, but they are not otherwise encoded, the code array just contains the raw float entries.
Subclassed by faiss::IndexIVFFlatDedup
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexIVFFlat(Index *quantizer, size_t d, size_t nlist_, MetricType = METRIC_L2)
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx) override
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
inline IndexIVFFlat()
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override¶
default implementation that calls encode_vectors
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)¶
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void train_residual(idx_t n, const float *x)¶
Sub-classes that encode the residuals can train their encoders here does nothing by default
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override¶
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
-
virtual void reconstruct(idx_t key, float *recons) const override¶
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)¶
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const override¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual size_t remove_ids(const IDSelector &sel) override¶
Dataset manipulation functions.
-
void check_compatible_for_merge(const IndexIVF &other) const¶
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(IndexIVF &other, idx_t add_id)¶
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal
-
virtual void copy_subset_to(IndexIVF &other, int subset_type, idx_t a1, idx_t a2) const¶
copy a subset of the entries index to the other index
if subset_type == 0: copies ids in [a1, a2) if subset_type == 1: copies ids if id % a1 == a2 if subset_type == 2: copies inverted lists such that a1 elements are left before and a2 elements are after
-
inline size_t get_list_size(size_t list_no) const¶
-
void make_direct_map(bool new_maintain_direct_map = true)¶
intialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)¶
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const¶
compute the number of bytes required to store list ids
Public Members
-
InvertedLists *invlists¶
Access to the actual data.
-
bool own_invlists¶
-
size_t code_size¶
code size per vector in bytes
-
size_t nprobe¶
number of probes at query time
-
size_t max_codes¶
max nb of codes to visit to do a query
-
int parallel_mode¶
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024¶
-
DirectMap direct_map¶
optional map that maps back ids to invlist entries. This enables reconstruct()
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
size_t nlist¶
number of possible key values
-
char quantizer_trains_alone¶
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields¶
whether object owns the quantizer (false by default)
-
ClusteringParameters cp¶
to override default clustering params
-
using idx_t = int64_t¶
-
struct IndexIVFFlatDedup : public faiss::IndexIVFFlat
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexIVFFlatDedup(Index *quantizer, size_t d, size_t nlist_, MetricType = METRIC_L2)
-
virtual void train(idx_t n, const float *x) override
also dedups the training set
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
implemented for all IndexIVF* classes
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override
not implemented
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v) override
not implemented
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
not implemented
-
inline IndexIVFFlatDedup()
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx) override
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override¶
removes all elements from the database.
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)¶
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void train_residual(idx_t n, const float *x)¶
Sub-classes that encode the residuals can train their encoders here does nothing by default
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override¶
assign the vectors, then call search_preassign
-
void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
-
virtual void reconstruct(idx_t key, float *recons) const override¶
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const override¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void check_compatible_for_merge(const IndexIVF &other) const¶
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(IndexIVF &other, idx_t add_id)¶
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal
-
virtual void copy_subset_to(IndexIVF &other, int subset_type, idx_t a1, idx_t a2) const¶
copy a subset of the entries index to the other index
if subset_type == 0: copies ids in [a1, a2) if subset_type == 1: copies ids if id % a1 == a2 if subset_type == 2: copies inverted lists such that a1 elements are left before and a2 elements are after
-
inline size_t get_list_size(size_t list_no) const¶
-
void make_direct_map(bool new_maintain_direct_map = true)¶
intialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)¶
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const¶
compute the number of bytes required to store list ids
Public Members
-
std::unordered_multimap<idx_t, idx_t> instances
Maps ids stored in the index to the ids of vectors that are the same. When a vector is unique, it does not appear in the instances map
-
InvertedLists *invlists¶
Access to the actual data.
-
bool own_invlists¶
-
size_t code_size¶
code size per vector in bytes
-
size_t nprobe¶
number of probes at query time
-
size_t max_codes¶
max nb of codes to visit to do a query
-
int parallel_mode¶
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024¶
-
DirectMap direct_map¶
optional map that maps back ids to invlist entries. This enables reconstruct()
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
size_t nlist¶
number of possible key values
-
char quantizer_trains_alone¶
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields¶
whether object owns the quantizer (false by default)
-
ClusteringParameters cp¶
to override default clustering params
-
using idx_t = int64_t¶
-
struct IndexIVFLocalSearchQuantizer : public faiss::IndexIVFAdditiveQuantizer
- #include <IndexIVFAdditiveQuantizer.h>
IndexIVF based on a residual quantizer. Stored vectors are approximated by residual quantization codes.
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexIVFLocalSearchQuantizer(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
IndexIVFLocalSearchQuantizer()
-
virtual ~IndexIVFLocalSearchQuantizer()
-
virtual void train_residual(idx_t n, const float *x) override
Sub-classes that encode the residuals can train their encoders here does nothing by default
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void sa_decode(idx_t n, const uint8_t *codes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override¶
default implementation that calls encode_vectors
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx)¶
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)¶
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override¶
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
-
virtual void reconstruct(idx_t key, float *recons) const override¶
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)¶
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const override¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const¶
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual size_t remove_ids(const IDSelector &sel) override¶
Dataset manipulation functions.
-
void check_compatible_for_merge(const IndexIVF &other) const¶
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(IndexIVF &other, idx_t add_id)¶
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal
-
virtual void copy_subset_to(IndexIVF &other, int subset_type, idx_t a1, idx_t a2) const¶
copy a subset of the entries index to the other index
if subset_type == 0: copies ids in [a1, a2) if subset_type == 1: copies ids if id % a1 == a2 if subset_type == 2: copies inverted lists such that a1 elements are left before and a2 elements are after
-
inline size_t get_list_size(size_t list_no) const¶
-
void make_direct_map(bool new_maintain_direct_map = true)¶
intialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)¶
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const¶
compute the number of bytes required to store list ids
Public Members
-
LocalSearchQuantizer lsq
The LSQ quantizer used to encode the vectors.
-
AdditiveQuantizer *aq
-
bool by_residual = true
-
int use_precomputed_table = 0
-
InvertedLists *invlists¶
Access to the actual data.
-
bool own_invlists¶
-
size_t code_size¶
code size per vector in bytes
-
size_t nprobe¶
number of probes at query time
-
size_t max_codes¶
max nb of codes to visit to do a query
-
int parallel_mode¶
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024¶
-
DirectMap direct_map¶
optional map that maps back ids to invlist entries. This enables reconstruct()
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
size_t nlist¶
number of possible key values
-
char quantizer_trains_alone¶
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields¶
whether object owns the quantizer (false by default)
-
ClusteringParameters cp¶
to override default clustering params
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexIVFLocalSearchQuantizerFastScan : public faiss::IndexIVFAdditiveQuantizerFastScan
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexIVFLocalSearchQuantizerFastScan(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_lsq2x4, int bbs = 32)
-
IndexIVFLocalSearchQuantizerFastScan()
-
~IndexIVFLocalSearchQuantizerFastScan()
-
void init(AdditiveQuantizer *aq, size_t nlist, MetricType metric, int bbs)
-
virtual void train_residual(idx_t n, const float *x) override
Sub-classes that encode the residuals can train their encoders here does nothing by default
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listno = false) const override
same as the regular IVFAQ encoder. The codes are not reorganized by blocks a that point
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
assign the vectors, then call search_preassign
-
virtual bool lookup_table_is_3d() const override
-
virtual void compute_LUT(size_t n, const float *x, const idx_t *coarse_ids, const float *coarse_dis, AlignedTable<float> &dis_tables, AlignedTable<float> &biases) const override
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void init_fastscan(size_t M, size_t nbits, size_t nlist, MetricType metric, int bbs)¶
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override¶
default implementation that calls encode_vectors
-
void compute_LUT_uint8(size_t n, const float *x, const idx_t *coarse_ids, const float *coarse_dis, AlignedTable<uint8_t> &dis_tables, AlignedTable<uint16_t> &biases, float *normalizers) const¶
-
template<bool is_max, class Scaler>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_1(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_2(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_10(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, size_t *ndis_out, size_t *nlist_out, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, size_t *ndis_out, size_t *nlist_out, const Scaler &scaler) const¶
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override¶
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
void reconstruct_orig_invlists()¶
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx)¶
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)¶
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs = false) const¶
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct(idx_t key, float *recons) const override¶
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)¶
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const override¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual size_t remove_ids(const IDSelector &sel) override¶
Dataset manipulation functions.
-
void check_compatible_for_merge(const IndexIVF &other) const¶
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(IndexIVF &other, idx_t add_id)¶
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal
-
virtual void copy_subset_to(IndexIVF &other, int subset_type, idx_t a1, idx_t a2) const¶
copy a subset of the entries index to the other index
if subset_type == 0: copies ids in [a1, a2) if subset_type == 1: copies ids if id % a1 == a2 if subset_type == 2: copies inverted lists such that a1 elements are left before and a2 elements are after
-
inline size_t get_list_size(size_t list_no) const¶
-
void make_direct_map(bool new_maintain_direct_map = true)¶
intialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)¶
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const¶
compute the number of bytes required to store list ids
Public Members
-
LocalSearchQuantizer lsq
-
AdditiveQuantizer *aq
-
bool rescale_norm = false
-
int norm_scale = 1
-
size_t max_train_points
-
int bbs¶
-
size_t M¶
-
size_t nbits¶
-
size_t ksub¶
-
size_t M2¶
-
int implem = 0¶
-
int skip = 0¶
-
bool by_residual = false¶
-
int qbs = 0¶
-
size_t qbs2 = 0¶
-
InvertedLists *orig_invlists = nullptr¶
orig’s inverted lists (for debugging)
-
InvertedLists *invlists¶
Access to the actual data.
-
bool own_invlists¶
-
size_t code_size¶
code size per vector in bytes
-
size_t nprobe¶
number of probes at query time
-
size_t max_codes¶
max nb of codes to visit to do a query
-
int parallel_mode¶
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024¶
-
DirectMap direct_map¶
optional map that maps back ids to invlist entries. This enables reconstruct()
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
size_t nlist¶
number of possible key values
-
char quantizer_trains_alone¶
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields¶
whether object owns the quantizer (false by default)
-
ClusteringParameters cp¶
to override default clustering params
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexIVFPQ : public faiss::IndexIVF
- #include <IndexIVFPQ.h>
Inverted file with Product Quantizer encoding. Each residual vector is encoded as a product quantizer code.
Subclassed by faiss::IndexIVFPQR
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexIVFPQ(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits_per_idx, MetricType metric = METRIC_L2)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx) override
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_core_o(idx_t n, const float *x, const idx_t *xids, float *residuals_2, const idx_t *precomputed_idx = nullptr)
same as add_core, also:
output 2nd level residuals if residuals_2 != NULL
accepts precomputed_idx = nullptr
-
virtual void train_residual(idx_t n, const float *x) override
trains the product quantizer
-
void train_residual_o(idx_t n, const float *x, float *residuals_2)
same as train_residual, also output 2nd level residuals
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
size_t find_duplicates(idx_t *ids, size_t *lims) const
Find exact duplicates in the dataset.
the duplicates are returned in pre-allocated arrays (see the max sizes).
- Parameters:
lims – limits between groups of duplicates (max size ntotal / 2 + 1)
ids – ids[lims[i]] : ids[lims[i+1]-1] is a group of duplicates (max size ntotal)
- Returns:
n number of groups found
-
void encode(idx_t key, const float *x, uint8_t *code) const
-
void encode_multiple(size_t n, idx_t *keys, const float *x, uint8_t *codes, bool compute_keys = false) const
Encode multiple vectors
- Parameters:
n – nb vectors to encode
keys – posting list ids for those vectors (size n)
x – vectors (size n * d)
codes – output codes (size n * code_size)
compute_keys – if false, assume keys are precomputed, otherwise compute them
-
void decode_multiple(size_t n, const idx_t *keys, const uint8_t *xcodes, float *x) const
inverse of encode_multiple
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
void precompute_table()
build precomputed table
-
IndexIVFPQ()
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override¶
default implementation that calls encode_vectors
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)¶
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override¶
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
-
virtual void reconstruct(idx_t key, float *recons) const override¶
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)¶
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const override¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual size_t remove_ids(const IDSelector &sel) override¶
Dataset manipulation functions.
-
void check_compatible_for_merge(const IndexIVF &other) const¶
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(IndexIVF &other, idx_t add_id)¶
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal
-
virtual void copy_subset_to(IndexIVF &other, int subset_type, idx_t a1, idx_t a2) const¶
copy a subset of the entries index to the other index
if subset_type == 0: copies ids in [a1, a2) if subset_type == 1: copies ids if id % a1 == a2 if subset_type == 2: copies inverted lists such that a1 elements are left before and a2 elements are after
-
inline size_t get_list_size(size_t list_no) const¶
-
void make_direct_map(bool new_maintain_direct_map = true)¶
intialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)¶
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const¶
compute the number of bytes required to store list ids
Public Members
-
bool by_residual
Encode residual or plain vector?
-
ProductQuantizer pq
produces the codes
-
bool do_polysemous_training
reorder PQ centroids after training?
-
PolysemousTraining *polysemous_training
if NULL, use default
-
size_t scan_table_threshold
use table computation or on-the-fly?
-
int polysemous_ht
Hamming thresh for polysemous filtering.
-
int use_precomputed_table
Precompute table that speed up query preprocessing at some memory cost (used only for by_residual with L2 metric)
-
AlignedTable<float> precomputed_table
if use_precompute_table size nlist * pq.M * pq.ksub
-
InvertedLists *invlists¶
Access to the actual data.
-
bool own_invlists¶
-
size_t code_size¶
code size per vector in bytes
-
size_t nprobe¶
number of probes at query time
-
size_t max_codes¶
max nb of codes to visit to do a query
-
int parallel_mode¶
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024¶
-
DirectMap direct_map¶
optional map that maps back ids to invlist entries. This enables reconstruct()
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
size_t nlist¶
number of possible key values
-
char quantizer_trains_alone¶
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields¶
whether object owns the quantizer (false by default)
-
ClusteringParameters cp¶
to override default clustering params
-
using idx_t = int64_t¶
-
struct IndexIVFPQFastScan : public faiss::IndexIVFFastScan
- #include <IndexIVFPQFastScan.h>
Fast scan version of IVFPQ. Works for 4-bit PQ for now.
The codes in the inverted lists are not stored sequentially but grouped in blocks of size bbs. This makes it possible to very quickly compute distances with SIMD instructions.
Implementations (implem): 0: auto-select implementation (default) 1: orig’s search, re-implemented 2: orig’s search, re-ordered by invlist 10: optimizer int16 search, collect results in heap, no qbs 11: idem, collect results in reservoir 12: optimizer int16 search, collect results in heap, uses qbs 13: idem, collect results in reservoir
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexIVFPQFastScan(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits, MetricType metric = METRIC_L2, int bbs = 32)
-
IndexIVFPQFastScan()
-
explicit IndexIVFPQFastScan(const IndexIVFPQ &orig, int bbs = 32)
-
virtual void train_residual(idx_t n, const float *x) override
Sub-classes that encode the residuals can train their encoders here does nothing by default
-
void precompute_table()
build precomputed table, possibly updating use_precomputed_table
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listno = false) const override
same as the regular IVFPQ encoder. The codes are not reorganized by blocks a that point
-
virtual bool lookup_table_is_3d() const override
-
virtual void compute_LUT(size_t n, const float *x, const idx_t *coarse_ids, const float *coarse_dis, AlignedTable<float> &dis_tables, AlignedTable<float> &biases) const override
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void init_fastscan(size_t M, size_t nbits, size_t nlist, MetricType metric, int bbs)¶
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override¶
default implementation that calls encode_vectors
-
void compute_LUT_uint8(size_t n, const float *x, const idx_t *coarse_ids, const float *coarse_dis, AlignedTable<uint8_t> &dis_tables, AlignedTable<uint16_t> &biases, float *normalizers) const¶
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override¶
assign the vectors, then call search_preassign
-
template<bool is_max, class Scaler>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_1(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_2(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_10(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, size_t *ndis_out, size_t *nlist_out, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, size_t *ndis_out, size_t *nlist_out, const Scaler &scaler) const¶
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override¶
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
void reconstruct_orig_invlists()¶
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx)¶
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)¶
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs = false) const¶
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct(idx_t key, float *recons) const override¶
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)¶
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const override¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual size_t remove_ids(const IDSelector &sel) override¶
Dataset manipulation functions.
-
void check_compatible_for_merge(const IndexIVF &other) const¶
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(IndexIVF &other, idx_t add_id)¶
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal
-
virtual void copy_subset_to(IndexIVF &other, int subset_type, idx_t a1, idx_t a2) const¶
copy a subset of the entries index to the other index
if subset_type == 0: copies ids in [a1, a2) if subset_type == 1: copies ids if id % a1 == a2 if subset_type == 2: copies inverted lists such that a1 elements are left before and a2 elements are after
-
inline size_t get_list_size(size_t list_no) const¶
-
void make_direct_map(bool new_maintain_direct_map = true)¶
intialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)¶
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const¶
compute the number of bytes required to store list ids
Public Members
-
ProductQuantizer pq
produces the codes
-
int use_precomputed_table = 0
precomputed tables management
-
AlignedTable<float> precomputed_table
if use_precompute_table size (nlist, pq.M, pq.ksub)
-
int bbs¶
-
size_t M¶
-
size_t nbits¶
-
size_t ksub¶
-
size_t M2¶
-
int implem = 0¶
-
int skip = 0¶
-
bool by_residual = false¶
-
int qbs = 0¶
-
size_t qbs2 = 0¶
-
InvertedLists *orig_invlists = nullptr¶
orig’s inverted lists (for debugging)
-
InvertedLists *invlists¶
Access to the actual data.
-
bool own_invlists¶
-
size_t code_size¶
code size per vector in bytes
-
size_t nprobe¶
number of probes at query time
-
size_t max_codes¶
max nb of codes to visit to do a query
-
int parallel_mode¶
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024¶
-
DirectMap direct_map¶
optional map that maps back ids to invlist entries. This enables reconstruct()
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
size_t nlist¶
number of possible key values
-
char quantizer_trains_alone¶
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields¶
whether object owns the quantizer (false by default)
-
ClusteringParameters cp¶
to override default clustering params
-
using idx_t = int64_t¶
-
struct IndexIVFPQR : public faiss::IndexIVFPQ
- #include <IndexIVFPQR.h>
Index with an additional level of PQ refinement
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexIVFPQR(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits_per_idx, size_t M_refine, size_t nbits_per_idx_refine)
-
virtual void reset() override
removes all elements from the database.
-
virtual size_t remove_ids(const IDSelector &sel) override
Dataset manipulation functions.
-
virtual void train_residual(idx_t n, const float *x) override
trains the two product quantizers
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
default implementation that calls encode_vectors
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx) override
same as add_with_ids, but optionally use the precomputed list ids
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual void merge_from(IndexIVF &other, idx_t add_id) override
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const override
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
IndexIVFPQR()
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override¶
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void add_core_o(idx_t n, const float *x, const idx_t *xids, float *residuals_2, const idx_t *precomputed_idx = nullptr)¶
same as add_core, also:
output 2nd level residuals if residuals_2 != NULL
accepts precomputed_idx = nullptr
-
void train_residual_o(idx_t n, const float *x, float *residuals_2)¶
same as train_residual, also output 2nd level residuals
-
size_t find_duplicates(idx_t *ids, size_t *lims) const¶
Find exact duplicates in the dataset.
the duplicates are returned in pre-allocated arrays (see the max sizes).
- Parameters:
lims – limits between groups of duplicates (max size ntotal / 2 + 1)
ids – ids[lims[i]] : ids[lims[i+1]-1] is a group of duplicates (max size ntotal)
- Returns:
n number of groups found
-
void encode_multiple(size_t n, idx_t *keys, const float *x, uint8_t *codes, bool compute_keys = false) const¶
Encode multiple vectors
- Parameters:
n – nb vectors to encode
keys – posting list ids for those vectors (size n)
x – vectors (size n * d)
codes – output codes (size n * code_size)
compute_keys – if false, assume keys are precomputed, otherwise compute them
-
void decode_multiple(size_t n, const idx_t *keys, const uint8_t *xcodes, float *x) const¶
inverse of encode_multiple
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs) const override¶
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
void precompute_table()¶
build precomputed table
-
virtual void train(idx_t n, const float *x) override¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)¶
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override¶
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
-
virtual void reconstruct(idx_t key, float *recons) const override¶
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)¶
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const override¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void check_compatible_for_merge(const IndexIVF &other) const¶
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void copy_subset_to(IndexIVF &other, int subset_type, idx_t a1, idx_t a2) const¶
copy a subset of the entries index to the other index
if subset_type == 0: copies ids in [a1, a2) if subset_type == 1: copies ids if id % a1 == a2 if subset_type == 2: copies inverted lists such that a1 elements are left before and a2 elements are after
-
inline size_t get_list_size(size_t list_no) const¶
-
void make_direct_map(bool new_maintain_direct_map = true)¶
intialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)¶
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const¶
compute the number of bytes required to store list ids
Public Members
-
ProductQuantizer refine_pq
3rd level quantizer
-
std::vector<uint8_t> refine_codes
corresponding codes
-
float k_factor
factor between k requested in search and the k requested from the IVFPQ
-
bool by_residual¶
Encode residual or plain vector?
-
ProductQuantizer pq¶
produces the codes
-
bool do_polysemous_training¶
reorder PQ centroids after training?
-
PolysemousTraining *polysemous_training¶
if NULL, use default
-
size_t scan_table_threshold¶
use table computation or on-the-fly?
-
int polysemous_ht¶
Hamming thresh for polysemous filtering.
-
int use_precomputed_table¶
Precompute table that speed up query preprocessing at some memory cost (used only for by_residual with L2 metric)
-
AlignedTable<float> precomputed_table¶
if use_precompute_table size nlist * pq.M * pq.ksub
-
InvertedLists *invlists¶
Access to the actual data.
-
bool own_invlists¶
-
size_t code_size¶
code size per vector in bytes
-
size_t nprobe¶
number of probes at query time
-
size_t max_codes¶
max nb of codes to visit to do a query
-
int parallel_mode¶
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024¶
-
DirectMap direct_map¶
optional map that maps back ids to invlist entries. This enables reconstruct()
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
size_t nlist¶
number of possible key values
-
char quantizer_trains_alone¶
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields¶
whether object owns the quantizer (false by default)
-
ClusteringParameters cp¶
to override default clustering params
-
using idx_t = int64_t¶
-
struct IndexIVFPQStats
- #include <IndexIVFPQ.h>
statistics are robust to internal threading, but not if IndexIVFPQ::search_preassigned is called by multiple threads
Public Functions
-
inline IndexIVFPQStats()
-
void reset()
Public Members
-
size_t nrefine
nb of refines (IVFPQR)
-
size_t n_hamming_pass
nb of passed Hamming distance tests (for polysemous)
-
size_t search_cycles
-
size_t refine_cycles
only for IVFPQR
-
inline IndexIVFPQStats()
-
struct IndexIVFResidualQuantizer : public faiss::IndexIVFAdditiveQuantizer
- #include <IndexIVFAdditiveQuantizer.h>
IndexIVF based on a residual quantizer. Stored vectors are approximated by residual quantization codes.
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexIVFResidualQuantizer(Index *quantizer, size_t d, size_t nlist, const std::vector<size_t> &nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
IndexIVFResidualQuantizer(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
-
IndexIVFResidualQuantizer()
-
virtual ~IndexIVFResidualQuantizer()
-
virtual void train_residual(idx_t n, const float *x) override
Sub-classes that encode the residuals can train their encoders here does nothing by default
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void sa_decode(idx_t n, const uint8_t *codes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override¶
default implementation that calls encode_vectors
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx)¶
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)¶
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override¶
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
-
virtual void reconstruct(idx_t key, float *recons) const override¶
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)¶
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const override¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const¶
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual size_t remove_ids(const IDSelector &sel) override¶
Dataset manipulation functions.
-
void check_compatible_for_merge(const IndexIVF &other) const¶
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(IndexIVF &other, idx_t add_id)¶
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal
-
virtual void copy_subset_to(IndexIVF &other, int subset_type, idx_t a1, idx_t a2) const¶
copy a subset of the entries index to the other index
if subset_type == 0: copies ids in [a1, a2) if subset_type == 1: copies ids if id % a1 == a2 if subset_type == 2: copies inverted lists such that a1 elements are left before and a2 elements are after
-
inline size_t get_list_size(size_t list_no) const¶
-
void make_direct_map(bool new_maintain_direct_map = true)¶
intialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)¶
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const¶
compute the number of bytes required to store list ids
Public Members
-
ResidualQuantizer rq
The residual quantizer used to encode the vectors.
-
AdditiveQuantizer *aq
-
bool by_residual = true
-
int use_precomputed_table = 0
-
InvertedLists *invlists¶
Access to the actual data.
-
bool own_invlists¶
-
size_t code_size¶
code size per vector in bytes
-
size_t nprobe¶
number of probes at query time
-
size_t max_codes¶
max nb of codes to visit to do a query
-
int parallel_mode¶
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024¶
-
DirectMap direct_map¶
optional map that maps back ids to invlist entries. This enables reconstruct()
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
size_t nlist¶
number of possible key values
-
char quantizer_trains_alone¶
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields¶
whether object owns the quantizer (false by default)
-
ClusteringParameters cp¶
to override default clustering params
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexIVFResidualQuantizerFastScan : public faiss::IndexIVFAdditiveQuantizerFastScan
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexIVFResidualQuantizerFastScan(Index *quantizer, size_t d, size_t nlist, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_lsq2x4, int bbs = 32)
-
IndexIVFResidualQuantizerFastScan()
-
~IndexIVFResidualQuantizerFastScan()
-
void init(AdditiveQuantizer *aq, size_t nlist, MetricType metric, int bbs)
-
virtual void train_residual(idx_t n, const float *x) override
Sub-classes that encode the residuals can train their encoders here does nothing by default
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listno = false) const override
same as the regular IVFAQ encoder. The codes are not reorganized by blocks a that point
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
assign the vectors, then call search_preassign
-
virtual bool lookup_table_is_3d() const override
-
virtual void compute_LUT(size_t n, const float *x, const idx_t *coarse_ids, const float *coarse_dis, AlignedTable<float> &dis_tables, AlignedTable<float> &biases) const override
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void init_fastscan(size_t M, size_t nbits, size_t nlist, MetricType metric, int bbs)¶
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override¶
default implementation that calls encode_vectors
-
void compute_LUT_uint8(size_t n, const float *x, const idx_t *coarse_ids, const float *coarse_dis, AlignedTable<uint8_t> &dis_tables, AlignedTable<uint16_t> &biases, float *normalizers) const¶
-
template<bool is_max, class Scaler>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_1(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_2(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_10(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, size_t *ndis_out, size_t *nlist_out, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, size_t *ndis_out, size_t *nlist_out, const Scaler &scaler) const¶
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override¶
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
void reconstruct_orig_invlists()¶
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx)¶
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)¶
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs = false) const¶
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct(idx_t key, float *recons) const override¶
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)¶
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const override¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual size_t remove_ids(const IDSelector &sel) override¶
Dataset manipulation functions.
-
void check_compatible_for_merge(const IndexIVF &other) const¶
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(IndexIVF &other, idx_t add_id)¶
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal
-
virtual void copy_subset_to(IndexIVF &other, int subset_type, idx_t a1, idx_t a2) const¶
copy a subset of the entries index to the other index
if subset_type == 0: copies ids in [a1, a2) if subset_type == 1: copies ids if id % a1 == a2 if subset_type == 2: copies inverted lists such that a1 elements are left before and a2 elements are after
-
inline size_t get_list_size(size_t list_no) const¶
-
void make_direct_map(bool new_maintain_direct_map = true)¶
intialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)¶
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const¶
compute the number of bytes required to store list ids
Public Members
-
ResidualQuantizer rq
-
AdditiveQuantizer *aq
-
bool rescale_norm = false
-
int norm_scale = 1
-
size_t max_train_points
-
int bbs¶
-
size_t M¶
-
size_t nbits¶
-
size_t ksub¶
-
size_t M2¶
-
int implem = 0¶
-
int skip = 0¶
-
bool by_residual = false¶
-
int qbs = 0¶
-
size_t qbs2 = 0¶
-
InvertedLists *orig_invlists = nullptr¶
orig’s inverted lists (for debugging)
-
InvertedLists *invlists¶
Access to the actual data.
-
bool own_invlists¶
-
size_t code_size¶
code size per vector in bytes
-
size_t nprobe¶
number of probes at query time
-
size_t max_codes¶
max nb of codes to visit to do a query
-
int parallel_mode¶
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024¶
-
DirectMap direct_map¶
optional map that maps back ids to invlist entries. This enables reconstruct()
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
size_t nlist¶
number of possible key values
-
char quantizer_trains_alone¶
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields¶
whether object owns the quantizer (false by default)
-
ClusteringParameters cp¶
to override default clustering params
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexIVFScalarQuantizer : public faiss::IndexIVF
- #include <IndexScalarQuantizer.h>
An IVF implementation where the components of the residuals are encoded with a scalar quantizer. All distance computations are asymmetric, so the encoded vectors are decoded and approximate distances are computed.
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexIVFScalarQuantizer(Index *quantizer, size_t d, size_t nlist, ScalarQuantizer::QuantizerType qtype, MetricType metric = METRIC_L2, bool encode_residual = true)
-
IndexIVFScalarQuantizer()
-
virtual void train_residual(idx_t n, const float *x) override
Sub-classes that encode the residuals can train their encoders here does nothing by default
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx) override
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const override
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override¶
default implementation that calls encode_vectors
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)¶
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override¶
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
-
virtual void reconstruct(idx_t key, float *recons) const override¶
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)¶
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const override¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual size_t remove_ids(const IDSelector &sel) override¶
Dataset manipulation functions.
-
void check_compatible_for_merge(const IndexIVF &other) const¶
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(IndexIVF &other, idx_t add_id)¶
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal
-
virtual void copy_subset_to(IndexIVF &other, int subset_type, idx_t a1, idx_t a2) const¶
copy a subset of the entries index to the other index
if subset_type == 0: copies ids in [a1, a2) if subset_type == 1: copies ids if id % a1 == a2 if subset_type == 2: copies inverted lists such that a1 elements are left before and a2 elements are after
-
inline size_t get_list_size(size_t list_no) const¶
-
void make_direct_map(bool new_maintain_direct_map = true)¶
intialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)¶
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const¶
compute the number of bytes required to store list ids
Public Members
-
ScalarQuantizer sq
-
bool by_residual
-
InvertedLists *invlists¶
Access to the actual data.
-
bool own_invlists¶
-
size_t code_size¶
code size per vector in bytes
-
size_t nprobe¶
number of probes at query time
-
size_t max_codes¶
max nb of codes to visit to do a query
-
int parallel_mode¶
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024¶
-
DirectMap direct_map¶
optional map that maps back ids to invlist entries. This enables reconstruct()
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
size_t nlist¶
number of possible key values
-
char quantizer_trains_alone¶
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields¶
whether object owns the quantizer (false by default)
-
ClusteringParameters cp¶
to override default clustering params
-
using idx_t = int64_t¶
-
struct IndexIVFSpectralHash : public faiss::IndexIVF
- #include <IndexIVFSpectralHash.h>
Inverted list that stores binary codes of size nbit. Before the binary conversion, the dimension of the vectors is transformed from dim d into dim nbit by vt (a random rotation by default).
Each coordinate is subtracted from a value determined by threshold_type, and split into intervals of size period. Half of the interval is a 0 bit, the other half a 1.
Public Types
-
enum ThresholdType
Values:
-
enumerator Thresh_global
global threshold at 0
-
enumerator Thresh_centroid
compare to centroid
-
enumerator Thresh_centroid_half
central interval around centroid
-
enumerator Thresh_median
median of training set
-
enumerator Thresh_global
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexIVFSpectralHash(Index *quantizer, size_t d, size_t nlist, int nbit, float period)
-
IndexIVFSpectralHash()
-
virtual void train_residual(idx_t n, const float *x) override
Sub-classes that encode the residuals can train their encoders here does nothing by default
-
virtual void encode_vectors(idx_t n, const float *x, const idx_t *list_nos, uint8_t *codes, bool include_listnos = false) const override
Encodes a set of vectors as they would appear in the inverted lists
- Parameters:
list_nos – inverted list ids as returned by the quantizer (size n). -1s are ignored.
codes – output codes, size n * code_size
include_listno – include the list ids in the code (in this case add ceil(log8(nlist)) to the code size)
-
virtual InvertedListScanner *get_InvertedListScanner(bool store_pairs) const override
Get a scanner for this index (store_pairs means ignore labels)
The default search implementation uses this to compute the distances
-
void replace_vt(VectorTransform *vt, bool own = false)
replace the vector transform for an empty (and possibly untrained) index
-
void replace_vt(IndexPreTransform *index, bool own = false)
convenience function to get the VT from an index constucted by an index_factory (should end in “LSH”)
-
~IndexIVFSpectralHash() override
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void train(idx_t n, const float *x) override¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override¶
default implementation that calls encode_vectors
-
virtual void add_core(idx_t n, const float *x, const idx_t *xids, const idx_t *precomputed_idx)¶
Implementation of vector addition where the vector assignments are predefined. The default implementation hands over the code extraction to encode_vectors.
- Parameters:
precomputed_idx – quantization indices for the input vectors (size n)
-
void add_sa_codes(idx_t n, const uint8_t *codes, const idx_t *xids)¶
Add vectors that are computed with the standalone codec
- Parameters:
codes – codes to add size n * sa_code_size()
xids – corresponding ids, size n
-
virtual void search_preassigned(idx_t n, const float *x, idx_t k, const idx_t *assign, const float *centroid_dis, float *distances, idx_t *labels, bool store_pairs, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
search a set of vectors, that are pre-quantized by the IVF quantizer. Fill in the corresponding heaps with the query results. The default implementation uses InvertedListScanners to do the search.
- Parameters:
n – nb of vectors to query
x – query vectors, size nx * d
assign – coarse quantization indices, size nx * nprobe
centroid_dis – distances to coarse centroids, size nx * nprobe
distance – output distances, size n * k
labels – output labels, size n * k
store_pairs – store inv list index + inv list offset instead in upper/lower 32 bit of result, instead of ids (used for reranking).
params – used to override the object’s search parameters
stats – search stats to be updated (can be null)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override¶
assign the vectors, then call search_preassign
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
void range_search_preassigned(idx_t nx, const float *x, float radius, const idx_t *keys, const float *coarse_dis, RangeSearchResult *result, bool store_pairs = false, const IVFSearchParameters *params = nullptr, IndexIVFStats *stats = nullptr) const¶
-
virtual void reconstruct(idx_t key, float *recons) const override¶
reconstruct a vector. Works only if maintain_direct_map is set to 1 or 2
-
virtual void update_vectors(int nv, const idx_t *idx, const float *v)¶
Update a subset of vectors.
The index must have a direct_map
- Parameters:
nv – nb of vectors to update
idx – vector indices to update, size nv
v – vectors of new values, size nv*d
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
Reconstruct a subset of the indexed vectors.
Overrides default implementation to bypass reconstruct() which requires direct_map to be maintained.
- Parameters:
i0 – first vector to reconstruct
ni – nb of vectors to reconstruct
recons – output array of reconstructed vectors, size ni * d
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const override¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
Overrides default implementation to avoid having to maintain direct_map and instead fetch the code offsets through the
store_pairsflag in search_preassigned().- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void reconstruct_from_offset(int64_t list_no, int64_t offset, float *recons) const¶
Reconstruct a vector given the location in terms of (inv list index + inv list offset) instead of the id.
Useful for reconstructing when the direct_map is not maintained and the inv list offset is computed by search_preassigned() with
store_pairsset.
-
virtual size_t remove_ids(const IDSelector &sel) override¶
Dataset manipulation functions.
-
void check_compatible_for_merge(const IndexIVF &other) const¶
check that the two indexes are compatible (ie, they are trained in the same way and have the same parameters). Otherwise throw.
-
virtual void merge_from(IndexIVF &other, idx_t add_id)¶
moves the entries from another dataset to self. On output, other is empty. add_id is added to all moved ids (for sequential ids, this would be this->ntotal
-
virtual void copy_subset_to(IndexIVF &other, int subset_type, idx_t a1, idx_t a2) const¶
copy a subset of the entries index to the other index
if subset_type == 0: copies ids in [a1, a2) if subset_type == 1: copies ids if id % a1 == a2 if subset_type == 2: copies inverted lists such that a1 elements are left before and a2 elements are after
-
inline size_t get_list_size(size_t list_no) const¶
-
void make_direct_map(bool new_maintain_direct_map = true)¶
intialize a direct map
- Parameters:
new_maintain_direct_map – if true, create a direct map, else clear it
-
void replace_invlists(InvertedLists *il, bool own = false)¶
replace the inverted lists, old one is deallocated if own_invlists
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)¶
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const¶
compute the number of bytes required to store list ids
Public Members
-
VectorTransform *vt
transformation from d to nbit dim
-
bool own_fields
own the vt
-
int nbit
nb of bits of the binary signature
-
float period
interval size for 0s and 1s
-
ThresholdType threshold_type
-
std::vector<float> trained
Trained threshold. size nlist * nbit or 0 if Thresh_global
-
InvertedLists *invlists¶
Access to the actual data.
-
bool own_invlists¶
-
size_t code_size¶
code size per vector in bytes
-
size_t nprobe¶
number of probes at query time
-
size_t max_codes¶
max nb of codes to visit to do a query
-
int parallel_mode¶
Parallel mode determines how queries are parallelized with OpenMP
0 (default): split over queries 1: parallelize over inverted lists 2: parallelize over both 3: split over queries with a finer granularity
PARALLEL_MODE_NO_HEAP_INIT: binary or with the previous to prevent the heap to be initialized and finalized
-
const int PARALLEL_MODE_NO_HEAP_INIT = 1024¶
-
DirectMap direct_map¶
optional map that maps back ids to invlist entries. This enables reconstruct()
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
size_t nlist¶
number of possible key values
-
char quantizer_trains_alone¶
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
ClusteringParameters cp¶
to override default clustering params
-
enum ThresholdType
-
struct IndexIVFStats
-
Public Members
-
size_t nq
-
size_t nlist
-
size_t ndis
-
size_t nheap_updates
-
double quantization_time
-
double search_time
-
size_t nq
-
struct IndexLattice : public faiss::Index
- #include <IndexLattice.h>
Index that encodes a vector with a series of Zn lattice quantizers
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexLattice(idx_t d, int nsq, int scale_nbit, int r2)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void add(idx_t n, const float *x) override
not implemented
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void reset() override
removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const¶
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
Public Members
-
int nsq
number of sub-vectors
-
size_t dsq
dimension of sub-vectors
-
ZnSphereCodecAlt zn_sphere_codec
the lattice quantizer
-
int scale_nbit
nb bits used to encode the scale, per subvector
-
int lattice_nbit
-
size_t code_size
total, in bytes
-
std::vector<float> trained
mins and maxes of the vector norms, per subquantizer
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
struct IndexLocalSearchQuantizer : public faiss::IndexAdditiveQuantizer
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexLocalSearchQuantizer(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
IndexLocalSearchQuantizer()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
reconstruction using the codec interface
-
virtual void reconstruct(idx_t key, float *recons) const override¶
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override¶
remove some ids. NB that Because of the structure of the indexing structure, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
LocalSearchQuantizer lsq
-
AdditiveQuantizer *aq
-
size_t code_size¶
-
std::vector<uint8_t> codes¶
encoded dataset, size ntotal * code_size
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexLocalSearchQuantizerFastScan : public faiss::IndexAdditiveQuantizerFastScan
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexLocalSearchQuantizerFastScan(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_lsq2x4, int bbs = 32)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
metric – metric type
search_type – AQ search type
-
IndexLocalSearchQuantizerFastScan()
-
void init(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override
-
virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
Decode a set of vectors.
NOTE: The codes in the IndexAdditiveQuantizerFastScan object are non- contiguous. But this method requires a contiguous representation.
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * code_size
x – output vectors, size n * d
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)¶
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void add(idx_t n, const float *x) override¶
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
template<bool is_max, class Scaler>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class Cfloat, class Scaler>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const Scaler &scaler) const¶
-
virtual void reconstruct(idx_t key, float *recons) const override¶
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
Public Members
-
LocalSearchQuantizer lsq
-
AdditiveQuantizer *aq
-
bool rescale_norm = true
-
int norm_scale = 1
-
size_t max_train_points = 0
-
int implem = 0¶
-
int skip = 0¶
-
int bbs¶
-
int qbs = 0¶
-
size_t M¶
-
size_t nbits¶
-
size_t ksub¶
-
size_t code_size¶
-
size_t ntotal2¶
-
size_t M2¶
-
AlignedTable<uint8_t> codes¶
-
const uint8_t *orig_codes = nullptr¶
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexLSH : public faiss::IndexFlatCodes
- #include <IndexLSH.h>
The sign of each vector component is put in a binary signature
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexLSH(idx_t d, int nbits, bool rotate_data = true, bool train_thresholds = false)
-
const float *apply_preprocess(idx_t n, const float *x) const
Preprocesses and resizes the input to the size required to binarize the data
- Parameters:
x – input vectors, size n * d
- Returns:
output vectors, size n * bits. May be the same pointer as x, otherwise it should be deleted by the caller
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
void transfer_thresholds(LinearTransform *vt)
transfer the thresholds to a pre-processing stage (and unset train_thresholds)
-
inline ~IndexLSH() override
-
IndexLSH()
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
reconstruction using the codec interface
-
virtual void reconstruct(idx_t key, float *recons) const override¶
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override¶
remove some ids. NB that Because of the structure of the indexing structure, the semantics of this operation are different from the usual ones: the new ids are shifted
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const¶
a FlatCodesDistanceComputer offers a distance_to_code method
-
inline virtual DistanceComputer *get_distance_computer() const override¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
int nbits
nb of bits per vector
-
bool rotate_data
whether to apply a random rotation to input
-
bool train_thresholds
whether we train thresholds or use 0
-
RandomRotationMatrix rrot
optional random rotation
-
std::vector<float> thresholds
thresholds to compare with
-
size_t code_size¶
-
std::vector<uint8_t> codes¶
encoded dataset, size ntotal * code_size
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
struct IndexNNDescent : public faiss::Index
- #include <IndexNNDescent.h>
The NNDescent index is a normal random-access index with an NNDescent link structure built on top
Subclassed by faiss::IndexNNDescentFlat
Public Types
-
using storage_idx_t = NNDescent::storage_idx_t
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
explicit IndexNNDescent(int d = 0, int K = 32, MetricType metric = METRIC_L2)
-
explicit IndexNNDescent(Index *storage, int K = 32)
-
~IndexNNDescent() override
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
entry point for search
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
using storage_idx_t = NNDescent::storage_idx_t
-
struct IndexNNDescentFlat : public faiss::IndexNNDescent
- #include <IndexNNDescent.h>
Flat index topped with with a NNDescent structure to access elements more efficiently.
Public Types
-
using storage_idx_t = NNDescent::storage_idx_t
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexNNDescentFlat()
-
IndexNNDescentFlat(int d, int K, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
entry point for search
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
using storage_idx_t = NNDescent::storage_idx_t
-
struct IndexNSG : public faiss::Index
- #include <IndexNSG.h>
The NSG index is a normal random-access index with a NSG link structure built on top
Subclassed by faiss::IndexNSGFlat, faiss::IndexNSGPQ, faiss::IndexNSGSQ
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
explicit IndexNSG(int d = 0, int R = 32, MetricType metric = METRIC_L2)
-
explicit IndexNSG(Index *storage, int R = 32)
-
~IndexNSG() override
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
entry point for search
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
NSG nsg
the link strcuture
-
bool own_fields
the sequential storage
-
Index *storage
-
bool is_built
the index is built or not
-
int GK
K of KNN graph for building.
-
char build_type
indicate how to build a knn graph
0: build NSG with brute force search
1: build NSG with NNDescent
-
int nndescent_S
parameters for nndescent
-
int nndescent_R
-
int nndescent_L
-
int nndescent_iter
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
struct IndexNSGFlat : public faiss::IndexNSG
- #include <IndexNSG.h>
Flat index topped with with a NSG structure to access elements more efficiently.
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexNSGFlat()
-
IndexNSGFlat(int d, int R, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
entry point for search
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
NSG nsg
the link strcuture
-
bool own_fields
the sequential storage
-
Index *storage
-
bool is_built
the index is built or not
-
int GK
K of KNN graph for building.
-
char build_type
indicate how to build a knn graph
0: build NSG with brute force search
1: build NSG with NNDescent
-
int nndescent_S
parameters for nndescent
-
int nndescent_R
-
int nndescent_L
-
int nndescent_iter
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
struct IndexNSGPQ : public faiss::IndexNSG
- #include <IndexNSG.h>
PQ index topped with with a NSG structure to access elements more efficiently.
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexNSGPQ()
-
IndexNSGPQ(int d, int pq_m, int M)
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
entry point for search
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
NSG nsg
the link strcuture
-
bool own_fields
the sequential storage
-
Index *storage
-
bool is_built
the index is built or not
-
int GK
K of KNN graph for building.
-
char build_type
indicate how to build a knn graph
0: build NSG with brute force search
1: build NSG with NNDescent
-
int nndescent_S
parameters for nndescent
-
int nndescent_R
-
int nndescent_L
-
int nndescent_iter
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
struct IndexNSGSQ : public faiss::IndexNSG
- #include <IndexNSG.h>
SQ index topped with with a NSG structure to access elements more efficiently.
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexNSGSQ()
-
IndexNSGSQ(int d, ScalarQuantizer::QuantizerType qtype, int M, MetricType metric = METRIC_L2)
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
virtual void train(idx_t n, const float *x) override
Trains the storage if needed.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
entry point for search
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reset() override
removes all elements from the database.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
NSG nsg
the link strcuture
-
bool own_fields
the sequential storage
-
Index *storage
-
bool is_built
the index is built or not
-
int GK
K of KNN graph for building.
-
char build_type
indicate how to build a knn graph
0: build NSG with brute force search
1: build NSG with NNDescent
-
int nndescent_S
parameters for nndescent
-
int nndescent_R
-
int nndescent_L
-
int nndescent_iter
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
struct IndexPQ : public faiss::IndexFlatCodes
- #include <IndexPQ.h>
Index based on a product quantizer. Stored vectors are approximated by PQ codes.
Public Types
-
enum Search_type_t
how to perform the search in search_core
Values:
-
enumerator ST_PQ
asymmetric product quantizer (default)
-
enumerator ST_HE
Hamming distance on codes.
-
enumerator ST_generalized_HE
nb of same codes
-
enumerator ST_SDC
symmetric product quantizer (SDC)
-
enumerator ST_polysemous
HE filter (using ht) + PQ combination.
-
enumerator ST_polysemous_generalize
Filter on generalized Hamming.
-
enumerator ST_PQ
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexPQ(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
IndexPQ()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
void search_core_polysemous(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const
-
void hamming_distance_histogram(idx_t n, const float *x, idx_t nb, const float *xb, int64_t *dist_histogram)
prepare query for a polysemous search, but instead of computing the result, just get the histogram of Hamming distances. May be computed on a provided dataset if xb != NULL
- Parameters:
dist_histogram – (M * nbits + 1)
-
void hamming_distance_table(idx_t n, const float *x, int32_t *dis) const
compute pairwise distances between queries and database
- Parameters:
n – nb of query vectors
x – query vector, size n * d
dis – output distances, size n * ntotal
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
reconstruction using the codec interface
-
virtual void reconstruct(idx_t key, float *recons) const override¶
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override¶
remove some ids. NB that Because of the structure of the indexing structure, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
ProductQuantizer pq
The product quantizer used to encode the vectors.
-
bool do_polysemous_training
false = standard PQ
-
PolysemousTraining polysemous_training
parameters used for the polysemous training
-
Search_type_t search_type
-
bool encode_signs
-
int polysemous_ht
Hamming threshold used for polysemy.
-
size_t code_size¶
-
std::vector<uint8_t> codes¶
encoded dataset, size ntotal * code_size
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
enum Search_type_t
-
struct IndexPQFastScan : public faiss::IndexFastScan
- #include <IndexPQFastScan.h>
Fast scan version of IndexPQ. Works for 4-bit PQ for now.
The codes are not stored sequentially but grouped in blocks of size bbs. This makes it possible to compute distances quickly with SIMD instructions.
Implementations: 12: blocked loop with internal loop on Q with qbs 13: same with reservoir accumulator to store results 14: no qbs with heap accumulator 15: no qbs with reservoir accumulator
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexPQFastScan(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, int bbs = 32)
-
IndexPQFastScan() = default
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override
-
virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
Decode a set of vectors.
NOTE: The codes in the IndexPQFastScan object are non-contiguous. But this method requires a contiguous representation.
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * code_size
x – output vectors, size n * d
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)¶
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override¶
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void add(idx_t n, const float *x) override¶
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
template<bool is_max, class Scaler>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class Cfloat, class Scaler>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const Scaler &scaler) const¶
-
virtual void reconstruct(idx_t key, float *recons) const override¶
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
Public Members
-
ProductQuantizer pq
-
int implem = 0¶
-
int skip = 0¶
-
int bbs¶
-
int qbs = 0¶
-
size_t M¶
-
size_t nbits¶
-
size_t ksub¶
-
size_t code_size¶
-
size_t ntotal2¶
-
size_t M2¶
-
AlignedTable<uint8_t> codes¶
-
const uint8_t *orig_codes = nullptr¶
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
struct IndexPQStats
- #include <IndexPQ.h>
statistics are robust to internal threading, but not if IndexPQ::search is called by multiple threads
Public Functions
-
inline IndexPQStats()
-
void reset()
Public Members
-
size_t nq
-
size_t ncode
-
size_t n_hamming_pass
-
inline IndexPQStats()
-
struct IndexPreTransform : public faiss::Index
- #include <IndexPreTransform.h>
Index that applies a LinearTransform transform on vectors before handing them over to a sub-index
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
explicit IndexPreTransform(Index *index)
! whether pointers are deleted in destructor
-
IndexPreTransform()
-
IndexPreTransform(VectorTransform *ltrans, Index *index)
ltrans is the last transform before the index
-
void prepend_transform(VectorTransform *ltrans)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids) override
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void reset() override
removes all elements from the database.
-
virtual size_t remove_ids(const IDSelector &sel) override
removes IDs from the index. Not supported by all indexes.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const override
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const override
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
const float *apply_chain(idx_t n, const float *x) const
apply the transforms in the chain. The returned float * may be equal to x, otherwise it should be deallocated.
-
void reverse_chain(idx_t n, const float *xt, float *x) const
Reverse the transforms in the chain. May not be implemented for all transforms in the chain or may return approximate results.
-
virtual DistanceComputer *get_distance_computer() const override
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
~IndexPreTransform() override
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
std::vector<VectorTransform*> chain
-
Index *index
! chain of tranforms
-
bool own_fields
! the sub-index
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
struct IndexRefine : public faiss::Index
- #include <IndexRefine.h>
Index that queries in a base_index (a fast one) and refines the results with an exact search, hopefully improving the results.
Subclassed by faiss::IndexRefineFlat
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexRefine()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
The sa_decode decodes from the index_refine, which is assumed to be more accurate
-
~IndexRefine() override
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
Public Members
-
Index *base_index
faster index to pre-select the vectors that should be filtered
-
Index *refine_index
refinement index
-
bool own_fields
should the base index be deallocated?
-
bool own_refine_index
same with the refinement index
-
float k_factor = 1
factor between k requested in search and the k requested from the base_index (should be >= 1)
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
struct IndexRefineFlat : public faiss::IndexRefine
- #include <IndexRefine.h>
Version where the refinement index is an IndexFlat. It has one additional constructor that takes a table of elements to add to the flat refinement index
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
explicit IndexRefineFlat(Index *base_index)
-
IndexRefineFlat(Index *base_index, const float *xb)
-
IndexRefineFlat()
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
The sa_decode decodes from the index_refine, which is assumed to be more accurate
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
Public Members
-
Index *base_index
faster index to pre-select the vectors that should be filtered
-
Index *refine_index
refinement index
-
bool own_fields
should the base index be deallocated?
-
bool own_refine_index
same with the refinement index
-
float k_factor = 1
factor between k requested in search and the k requested from the base_index (should be >= 1)
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
template<typename IndexT>
class IndexReplicasTemplate : public ThreadedIndex<IndexT> - #include <IndexReplicas.h>
Takes individual faiss::Index instances, and splits queries for sending to each Index instance, and joins the results together when done. Each index is managed by a separate CPU thread.
Public Types
-
using idx_t = typename IndexT::idx_t
-
using component_t = typename IndexT::component_t
-
using distance_t = typename IndexT::distance_t
Public Functions
-
explicit IndexReplicasTemplate(bool threaded = true)
The dimension that all sub-indices must share will be the dimension of the first sub-index added
- Parameters:
threaded – do we use one thread per sub-index or do queries sequentially?
-
explicit IndexReplicasTemplate(idx_t d, bool threaded = true)
- Parameters:
d – the dimension that all sub-indices must share
threaded – do we use one thread per sub index or do queries sequentially?
-
explicit IndexReplicasTemplate(int d, bool threaded = true)
int version due to the implicit bool conversion ambiguity of int as dimension
-
inline void add_replica(IndexT *index)
Alias for addIndex()
-
inline void remove_replica(IndexT *index)
Alias for removeIndex()
-
void train(idx_t n, const component_t *x) override
faiss::Index API All indices receive the same call
-
void add(idx_t n, const component_t *x) override
faiss::Index API All indices receive the same call
-
void search(idx_t n, const component_t *x, idx_t k, distance_t *distances, idx_t *labels) const override
faiss::Index API Query is partitioned into a slice for each sub-index split by ceil(n / #indices) for our sub-indices
-
void reconstruct(idx_t, component_t *v) const override
reconstructs from the first index
-
void syncWithSubIndexes()
Synchronize the top-level index (IndexShards) with data in the sub-indices
-
using idx_t = typename IndexT::idx_t
-
struct IndexResidualQuantizer : public faiss::IndexAdditiveQuantizer
- #include <IndexAdditiveQuantizer.h>
Index based on a residual quantizer. Stored vectors are approximated by residual quantization codes. Can also be used as a codec
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexResidualQuantizer(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
IndexResidualQuantizer(int d, const std::vector<size_t> &nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_decompress)
-
IndexResidualQuantizer()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
reconstruction using the codec interface
-
virtual void reconstruct(idx_t key, float *recons) const override¶
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override¶
remove some ids. NB that Because of the structure of the indexing structure, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
ResidualQuantizer rq
The residual quantizer used to encode the vectors.
-
AdditiveQuantizer *aq
-
size_t code_size¶
-
std::vector<uint8_t> codes¶
encoded dataset, size ntotal * code_size
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexResidualQuantizerFastScan : public faiss::IndexAdditiveQuantizerFastScan
- #include <IndexAdditiveQuantizerFastScan.h>
Index based on a residual quantizer. Stored vectors are approximated by residual quantization codes. Can also be used as a codec
Public Types
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexResidualQuantizerFastScan(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2, Search_type_t search_type = AdditiveQuantizer::ST_norm_rq2x4, int bbs = 32)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
metric – metric type
search_type – AQ search type
-
IndexResidualQuantizerFastScan()
-
void init(AdditiveQuantizer *aq, MetricType metric = METRIC_L2, int bbs = 32)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
void estimate_norm_scale(idx_t n, const float *x)
-
virtual void compute_codes(uint8_t *codes, idx_t n, const float *x) const override
-
virtual void compute_float_LUT(float *lut, idx_t n, const float *x) const override
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
Decode a set of vectors.
NOTE: The codes in the IndexAdditiveQuantizerFastScan object are non- contiguous. But this method requires a contiguous representation.
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * code_size
x – output vectors, size n * d
-
void init_fastscan(int d, size_t M, size_t nbits, MetricType metric, int bbs)¶
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void add(idx_t n, const float *x) override¶
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
template<bool is_max, class Scaler>
void search_dispatch_implem(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class Cfloat, class Scaler>
void search_implem_234(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_12(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const Scaler &scaler) const¶
-
template<class C, class Scaler>
void search_implem_14(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, int impl, const Scaler &scaler) const¶
-
virtual void reconstruct(idx_t key, float *recons) const override¶
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
Public Members
-
ResidualQuantizer rq
The residual quantizer used to encode the vectors.
-
AdditiveQuantizer *aq
-
bool rescale_norm = true
-
int norm_scale = 1
-
size_t max_train_points = 0
-
int implem = 0¶
-
int skip = 0¶
-
int bbs¶
-
int qbs = 0¶
-
size_t M¶
-
size_t nbits¶
-
size_t ksub¶
-
size_t code_size¶
-
size_t ntotal2¶
-
size_t M2¶
-
AlignedTable<uint8_t> codes¶
-
const uint8_t *orig_codes = nullptr¶
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using Search_type_t = AdditiveQuantizer::Search_type_t
-
struct IndexScalarQuantizer : public faiss::IndexFlatCodes
- #include <IndexScalarQuantizer.h>
Flat index built on a scalar quantizer.
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
IndexScalarQuantizer(int d, ScalarQuantizer::QuantizerType qtype, MetricType metric = METRIC_L2)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
IndexScalarQuantizer()
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual FlatCodesDistanceComputer *get_FlatCodesDistanceComputer() const override
a FlatCodesDistanceComputer offers a distance_to_code method
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const override
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const override
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
virtual void reset() override¶
removes all elements from the database.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const override¶
reconstruction using the codec interface
-
virtual void reconstruct(idx_t key, float *recons) const override¶
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual size_t sa_code_size() const override¶
size of the produced codes in bytes
-
virtual size_t remove_ids(const IDSelector &sel) override¶
remove some ids. NB that Because of the structure of the indexing structure, the semantics of this operation are different from the usual ones: the new ids are shifted
-
inline virtual DistanceComputer *get_distance_computer() const override¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
Public Members
-
ScalarQuantizer sq
Used to encode the vectors.
-
size_t code_size¶
-
std::vector<uint8_t> codes¶
encoded dataset, size ntotal * code_size
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
template<typename IndexT>
struct IndexShardsTemplate : public ThreadedIndex<IndexT> - #include <IndexShards.h>
Index that concatenates the results from several sub-indexes
Public Types
-
using idx_t = typename IndexT::idx_t
-
using component_t = typename IndexT::component_t
-
using distance_t = typename IndexT::distance_t
Public Functions
-
explicit IndexShardsTemplate(bool threaded = false, bool successive_ids = true)
The dimension that all sub-indices must share will be the dimension of the first sub-index added
- Parameters:
threaded – do we use one thread per sub_index or do queries sequentially?
successive_ids – should we shift the returned ids by the size of each sub-index or return them as they are?
-
explicit IndexShardsTemplate(idx_t d, bool threaded = false, bool successive_ids = true)
- Parameters:
threaded – do we use one thread per sub_index or do queries sequentially?
successive_ids – should we shift the returned ids by the size of each sub-index or return them as they are?
-
explicit IndexShardsTemplate(int d, bool threaded = false, bool successive_ids = true)
int version due to the implicit bool conversion ambiguity of int as dimension
-
inline void add_shard(IndexT *index)
Alias for addIndex()
-
inline void remove_shard(IndexT *index)
Alias for removeIndex()
-
void add(idx_t n, const component_t *x) override
supported only for sub-indices that implement add_with_ids
-
void add_with_ids(idx_t n, const component_t *x, const idx_t *xids) override
Cases (successive_ids, xids):
true, non-NULL ERROR: it makes no sense to pass in ids and request them to be shifted
true, NULL OK, but should be called only once (calls add() on sub-indexes).
false, non-NULL OK: will call add_with_ids with passed in xids distributed evenly over shards
false, NULL OK: will call add_with_ids on each sub-index, starting at ntotal
-
void search(idx_t n, const component_t *x, idx_t k, distance_t *distances, idx_t *labels) const override
-
void train(idx_t n, const component_t *x) override
-
void syncWithSubIndexes()
Synchronize the top-level index (IndexShards) with data in the sub-indices
Public Members
-
bool successive_ids
-
using idx_t = typename IndexT::idx_t
-
struct IndexSplitVectors : public faiss::Index
- #include <MetaIndexes.h>
splits input vectors in segments and assigns each segment to a sub-index used to distribute a MultiIndexQuantizer
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
explicit IndexSplitVectors(idx_t d, bool threaded = false)
sum of dimensions seen so far
-
void add_sub_index(Index*)
-
void sync_with_sub_indexes()
-
virtual void add(idx_t n, const float *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1 This function slices the input vectors in chunks smaller than blocksize_add and calls add_core.
- Parameters:
x – input matrix, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void reset() override
removes all elements from the database.
-
~IndexSplitVectors() override
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const¶
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
using idx_t = int64_t¶
-
struct IntersectionCriterion : public faiss::AutoTuneCriterion
-
Public Functions
-
virtual double evaluate(const float *D, const idx_t *I) const override
Evaluate the criterion.
- Parameters:
D – size nq * nnn
I – size nq * nnn
- Returns:
the criterion, between 0 and 1. Larger is better.
-
inline ~IntersectionCriterion() override
-
void set_groundtruth(int gt_nnn, const float *gt_D_in, const idx_t *gt_I_in)
Intitializes the gt_D and gt_I vectors. Must be called before evaluating
- Parameters:
gt_D_in – size nq * gt_nnn
gt_I_in – size nq * gt_nnn
Public Members
-
idx_t R
-
idx_t nq
nb of queries this criterion is evaluated on
-
idx_t nnn
nb of NNs that the query should request
-
idx_t gt_nnn
nb of GT NNs required to evaluate criterion
-
std::vector<float> gt_D
Ground-truth distances (size nq * gt_nnn)
-
std::vector<idx_t> gt_I
Ground-truth indexes (size nq * gt_nnn)
-
virtual double evaluate(const float *D, const idx_t *I) const override
-
struct InvertedLists
- #include <InvertedLists.h>
Table of inverted lists multithreading rules:
concurrent read accesses are allowed
concurrent update accesses are allowed
for resize and add_entries, only concurrent access to different lists are allowed
Subclassed by faiss::ArrayInvertedLists, faiss::BlockInvertedLists, faiss::OnDiskInvertedLists, faiss::ReadOnlyInvertedLists
Public Functions
-
InvertedLists(size_t nlist, size_t code_size)
-
virtual size_t list_size(size_t list_no) const = 0
get the size of a list
-
virtual const uint8_t *get_codes(size_t list_no) const = 0
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const = 0
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const
release codes returned by get_codes (default implementation is nop
-
virtual void release_ids(size_t list_no, const idx_t *ids) const
release ids returned by get_ids
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code)
add one entry to an inverted list
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) = 0
-
virtual void update_entry(size_t list_no, size_t offset, idx_t id, const uint8_t *code)
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) = 0
-
virtual void resize(size_t list_no, size_t new_size) = 0
-
virtual void reset()
-
void merge_from(InvertedLists *oivf, size_t add_id)
move all entries from oivf (empty on output)
-
virtual ~InvertedLists()
-
double imbalance_factor() const
1= perfectly balanced, >1: imbalanced
-
void print_stats() const
display some stats about the inverted lists
-
size_t compute_ntotal() const
sum up list sizes
Public Members
-
size_t nlist
number of possible key values
-
size_t code_size
code size per vector in bytes
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
struct ScopedCodes
Public Functions
-
inline ScopedCodes(const InvertedLists *il, size_t list_no)
-
inline ScopedCodes(const InvertedLists *il, size_t list_no, size_t offset)
-
inline const uint8_t *get()
-
inline ~ScopedCodes()
-
inline ScopedCodes(const InvertedLists *il, size_t list_no)
-
struct ScopedIds
Public Functions
-
inline ScopedIds(const InvertedLists *il, size_t list_no)
-
inline const idx_t *get()
-
inline idx_t operator[](size_t i) const
-
inline ~ScopedIds()
-
inline ScopedIds(const InvertedLists *il, size_t list_no)
-
struct InvertedListScanner
- #include <IndexIVF.h>
Object that handles a query. The inverted lists to scan are provided externally. The object has a lot of state, but distance_to_code and scan_codes can be called in multiple threads
Public Functions
-
virtual void set_query(const float *query_vector) = 0
from now on we handle this query.
-
virtual void set_list(idx_t list_no, float coarse_dis) = 0
following codes come from this inverted list
-
virtual float distance_to_code(const uint8_t *code) const = 0
compute a single query-to-code distance
-
virtual size_t scan_codes(size_t n, const uint8_t *codes, const idx_t *ids, float *distances, idx_t *labels, size_t k) const
scan a set of codes, compute distances to current query and update heap of results if necessary. Default implemetation calls distance_to_code.
- Parameters:
n – number of codes to scan
codes – codes to scan (n * code_size)
ids – corresponding ids (ignored if store_pairs)
distances – heap distances (size k)
labels – heap labels (size k)
k – heap size
- Returns:
number of heap updates performed
-
virtual void scan_codes_range(size_t n, const uint8_t *codes, const idx_t *ids, float radius, RangeQueryResult &result) const
scan a set of codes, compute distances to current query and update results if distances are below radius
(default implementation fails)
-
inline virtual ~InvertedListScanner()
Public Members
-
idx_t list_no = -1
remember current list
-
bool keep_max = false
keep maximum instead of minimum
-
bool store_pairs = false
store positions in invlists rather than labels
-
size_t code_size = 0
used in default implementation of scan_codes
-
virtual void set_query(const float *query_vector) = 0
-
struct InvertedListsIOHook
- #include <InvertedListsIOHook.h>
Callbacks to handle other types of InvertedList objects.
The callbacks should be registered with add_callback before calling read_index or read_InvertedLists. The callbacks for OnDiskInvertedLists are registrered by default. The invlist type is identified by:
the key (a fourcc) at read time
the class name (as given by typeid.name) at write time
Subclassed by faiss::BlockInvertedListsIOHook, faiss::OnDiskInvertedListsIOHook
Public Functions
-
InvertedListsIOHook(const std::string &key, const std::string &classname)
-
virtual void write(const InvertedLists *ils, IOWriter *f) const = 0
write the index to the IOWriter (including the fourcc)
-
virtual InvertedLists *read(IOReader *f, int io_flags) const = 0
called when the fourcc matches this class’s fourcc
-
virtual InvertedLists *read_ArrayInvertedLists(IOReader *f, int io_flags, size_t nlist, size_t code_size, const std::vector<size_t> &sizes) const
read from a ArrayInvertedLists into this invertedlist type. For this to work, the callback has to be enabled and the io_flag has to be set to IO_FLAG_SKIP_IVF_DATA | (16 upper bits of the fourcc)
(default implementation fails)
-
inline virtual ~InvertedListsIOHook()
Public Members
-
const std::string key
string version of the fourcc
-
const std::string classname
typeid.name
Public Static Functions
-
static void add_callback(InvertedListsIOHook*)
-
static void print_callbacks()
-
static InvertedListsIOHook *lookup(int h)
-
static InvertedListsIOHook *lookup_classname(const std::string &classname)
-
struct ITQMatrix : public faiss::LinearTransform
- #include <VectorTransform.h>
ITQ implementation from
Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval,
Yunchao Gong, Svetlana Lazebnik, Albert Gordo, Florent Perronnin, PAMI’12.
Public Functions
-
explicit ITQMatrix(int d = 0)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors. Does nothing by default.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
void transform_transpose(idx_t n, const float *y, float *x) const
compute x = A^T * (x - b) is reverse transform if A has orthonormal lines
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
works only if is_orthonormal
-
void set_is_orthonormal()
compute A^T * A to set the is_orthonormal flag
-
void print_if_verbose(const char *name, const std::vector<double> &mat, int n, int d) const
-
float *apply(idx_t n, const float *x) const
apply the random rotation, return new allocated matrix
- Parameters:
x – size n * d_in
- Returns:
size n * d_out
Public Members
-
int max_iter
-
int seed
-
std::vector<double> init_rotation
-
bool have_bias
-
bool is_orthonormal
! whether to use the bias term
check if matrix A is orthonormal (enables reverse_transform)
-
std::vector<float> A
Transformation matrix, size d_out * d_in.
-
std::vector<float> b
bias vector, size d_out
-
bool verbose
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit ITQMatrix(int d = 0)
-
struct ITQTransform : public faiss::VectorTransform
- #include <VectorTransform.h>
The full ITQ transform, including normalizations and PCA transformation
Public Functions
-
explicit ITQTransform(int d_in = 0, int d_out = 0, bool do_pca = false)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors. Does nothing by default.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
float *apply(idx_t n, const float *x) const
apply the random rotation, return new allocated matrix
- Parameters:
x – size n * d_in
- Returns:
size n * d_out
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const
reverse transformation. May not be implemented or may return approximate result
Public Members
-
std::vector<float> mean
-
bool do_pca
-
ITQMatrix itq
-
int max_train_per_dim
max training points per dimension
-
LinearTransform pca_then_itq
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit ITQTransform(int d_in = 0, int d_out = 0, bool do_pca = false)
-
struct IVFFastScanStats
Public Functions
-
inline double Mcy_at(int i)
-
inline double Mcy_reservoir_at(int i)
-
inline IVFFastScanStats()
-
inline void reset()
Public Members
-
uint64_t times[10]
-
uint64_t t_compute_distance_tables
-
uint64_t t_round
-
uint64_t t_copy_pack
-
uint64_t t_scan
-
uint64_t t_to_flat
-
uint64_t reservoir_times[4]
-
double t_aq_encode
-
double t_aq_norm_encode
-
inline double Mcy_at(int i)
-
struct IVFPQSearchParameters : public faiss::IVFSearchParameters
Public Functions
-
inline IVFPQSearchParameters()
-
inline ~IVFPQSearchParameters()
-
inline IVFPQSearchParameters()
-
struct IVFSearchParameters
Subclassed by faiss::IVFPQSearchParameters
Public Functions
-
inline IVFSearchParameters()
-
inline virtual ~IVFSearchParameters()
Public Members
-
size_t nprobe
number of probes at query time
-
size_t max_codes
max nb of codes to visit to do a query
-
inline IVFSearchParameters()
-
struct Level1Quantizer
- #include <IndexIVF.h>
Encapsulates a quantizer object for the IndexIVF
The class isolates the fields that are independent of the storage of the lists (especially training)
Subclassed by faiss::IndexIVF
Public Functions
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
Trains the quantizer and calls train_residual to train sub-quantizers.
-
size_t coarse_code_size() const
compute the number of bytes required to store list ids
-
Level1Quantizer(Index *quantizer, size_t nlist)
-
Level1Quantizer()
-
~Level1Quantizer()
Public Members
-
Index *quantizer
quantizer that maps vectors to inverted lists
-
size_t nlist
number of possible key values
-
char quantizer_trains_alone
= 0: use the quantizer as index in a kmeans training = 1: just pass on the training set to the train() of the quantizer = 2: kmeans training on a flat index + add the centroids to the quantizer
-
bool own_fields
whether object owns the quantizer (false by default)
-
ClusteringParameters cp
to override default clustering params
-
Index *clustering_index
to override index used during clustering
-
void train_q1(size_t n, const float *x, bool verbose, MetricType metric_type)
-
struct LinearTransform : public faiss::VectorTransform
- #include <VectorTransform.h>
Generic linear transformation, with bias term applied on output y = A * x + b
Subclassed by faiss::ITQMatrix, faiss::OPQMatrix, faiss::PCAMatrix, faiss::RandomRotationMatrix
Public Functions
-
explicit LinearTransform(int d_in = 0, int d_out = 0, bool have_bias = false)
both d_in > d_out and d_out < d_in are supported
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
void transform_transpose(idx_t n, const float *y, float *x) const
compute x = A^T * (x - b) is reverse transform if A has orthonormal lines
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
works only if is_orthonormal
-
void set_is_orthonormal()
compute A^T * A to set the is_orthonormal flag
-
void print_if_verbose(const char *name, const std::vector<double> &mat, int n, int d) const
-
inline ~LinearTransform() override
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors. Does nothing by default.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
float *apply(idx_t n, const float *x) const
apply the random rotation, return new allocated matrix
- Parameters:
x – size n * d_in
- Returns:
size n * d_out
Public Members
-
bool have_bias
-
bool is_orthonormal
! whether to use the bias term
check if matrix A is orthonormal (enables reverse_transform)
-
std::vector<float> A
Transformation matrix, size d_out * d_in.
-
std::vector<float> b
bias vector, size d_out
-
bool verbose
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit LinearTransform(int d_in = 0, int d_out = 0, bool have_bias = false)
-
struct LocalSearchCoarseQuantizer : public faiss::AdditiveCoarseQuantizer
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
LocalSearchCoarseQuantizer(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
LocalSearchCoarseQuantizer()
-
virtual void add(idx_t n, const float *x) override
N/A.
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void reset() override
N/A.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
LocalSearchQuantizer lsq
The residual quantizer used to encode the vectors.
-
AdditiveQuantizer *aq
-
std::vector<float> centroid_norms
norms of centroids, useful for knn-search
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
struct MaskedInvertedLists : public faiss::ReadOnlyInvertedLists
- #include <InvertedLists.h>
use the first inverted lists if they are non-empty otherwise use the second
This is useful if il1 has a few inverted lists that are too long, and that il0 has replacement lists for those, with empty lists for the others.
Public Functions
-
MaskedInvertedLists(const InvertedLists *il0, const InvertedLists *il1)
-
virtual size_t list_size(size_t list_no) const override
get the size of a list
-
virtual const uint8_t *get_codes(size_t list_no) const override
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const override
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const override
release codes returned by get_codes (default implementation is nop
-
virtual void release_ids(size_t list_no, const idx_t *ids) const override
release ids returned by get_ids
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const override
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const override
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const override
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void resize(size_t list_no, size_t new_size) override
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code)
add one entry to an inverted list
-
virtual void update_entry(size_t list_no, size_t offset, idx_t id, const uint8_t *code)
-
virtual void reset()
-
void merge_from(InvertedLists *oivf, size_t add_id)
move all entries from oivf (empty on output)
-
double imbalance_factor() const
1= perfectly balanced, >1: imbalanced
-
void print_stats() const
display some stats about the inverted lists
-
size_t compute_ntotal() const
sum up list sizes
Public Members
-
const InvertedLists *il0
-
const InvertedLists *il1
-
size_t nlist
number of possible key values
-
size_t code_size
code size per vector in bytes
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
MaskedInvertedLists(const InvertedLists *il0, const InvertedLists *il1)
-
struct MatrixStats
- #include <MatrixStats.h>
Reports some statistics on a dataset and comments on them.
It is a class rather than a function so that all stats can also be accessed from code
Public Functions
-
MatrixStats(size_t n, size_t d, const float *x)
-
void do_comment(const char *fmt, ...)
Public Members
-
std::string comments
-
size_t n
-
size_t d
-
size_t n_collision
-
size_t n_valid
-
size_t n0
-
double min_norm2
-
double max_norm2
-
std::vector<PerDimStats> per_dim_stats
-
std::unordered_map<uint64_t, Occurrence> occurrences
-
char *buf
-
size_t nbuf
-
struct Occurrence
Public Members
-
size_t first
-
size_t count
-
size_t first
-
struct PerDimStats
Public Functions
-
PerDimStats()
-
void add(float x)
-
void compute_mean_std()
Public Members
-
size_t n
-
size_t n_nan
-
size_t n_inf
-
size_t n0
-
float min
-
float max
-
double sum
-
double sum2
-
size_t n_valid
-
double mean
-
double stddev
-
PerDimStats()
-
MatrixStats(size_t n, size_t d, const float *x)
-
struct MultiIndexQuantizer : public faiss::Index
- #include <IndexPQ.h>
Quantizer where centroids are virtual: they are the Cartesian product of sub-centroids.
Subclassed by faiss::MultiIndexQuantizer2
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
MultiIndexQuantizer(int d, size_t M, size_t nbits)
number of bit per subvector index
- Parameters:
d – dimension of the input vectors
M – number of subquantizers
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void add(idx_t n, const float *x) override
add and reset will crash at runtime
-
virtual void reset() override
removes all elements from the database.
-
inline MultiIndexQuantizer()
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
ProductQuantizer pq
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
struct MultiIndexQuantizer2 : public faiss::MultiIndexQuantizer
- #include <IndexPQ.h>
MultiIndexQuantizer where the PQ assignmnet is performed by sub-indexes
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
MultiIndexQuantizer2(int d, size_t M, size_t nbits, Index **indexes)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void add(idx_t n, const float *x) override
add and reset will crash at runtime
-
virtual void reset() override
removes all elements from the database.
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
-
using idx_t = int64_t¶
-
struct NormalizationTransform : public faiss::VectorTransform
- #include <VectorTransform.h>
per-vector normalization
Public Functions
-
explicit NormalizationTransform(int d, float norm = 2.0)
-
NormalizationTransform()
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
Identity transform since norm is not revertible.
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors. Does nothing by default.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
float *apply(idx_t n, const float *x) const
apply the random rotation, return new allocated matrix
- Parameters:
x – size n * d_in
- Returns:
size n * d_out
Public Members
-
float norm
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit NormalizationTransform(int d, float norm = 2.0)
-
struct OnDiskInvertedLists : public faiss::InvertedLists
- #include <OnDiskInvertedLists.h>
On-disk storage of inverted lists.
The data is stored in a mmapped chunk of memory (base ptointer ptr, size totsize). Each list is a range of memory that contains (object List) that contains:
uint8_t codes[capacity * code_size]
followed by idx_t ids[capacity]
in each of the arrays, the size <= capacity first elements are used, the rest is not initialized.
Addition and resize are supported by:
roundind up the capacity of the lists to a power of two
maintaining a list of empty slots, sorted by size.
resizing the mmapped block is adjusted as needed.
An OnDiskInvertedLists is compact if the size == capacity for all lists and there are no available slots.
Addition to the invlists is slow. For incremental add it is better to use a default ArrayInvertedLists object and convert it to an OnDisk with merge_from.
When it is known that a set of lists will be accessed, it is useful to call prefetch_lists, that launches a set of threads to read the lists in parallel.
Public Functions
-
OnDiskInvertedLists(size_t nlist, size_t code_size, const char *filename)
are inverted lists mapped read-only
-
virtual size_t list_size(size_t list_no) const override
get the size of a list
-
virtual const uint8_t *get_codes(size_t list_no) const override
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const override
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void resize(size_t list_no, size_t new_size) override
-
size_t merge_from(const InvertedLists **ils, int n_il, bool verbose = false)
-
size_t merge_from_1(const InvertedLists *il, bool verbose = false)
same as merge_from for a single invlist
-
void crop_invlists(size_t l0, size_t l1)
restrict the inverted lists to l0:l1 without touching the mmapped region
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const override
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
~OnDiskInvertedLists() override
-
void do_mmap()
-
void update_totsize(size_t new_totsize)
-
void resize_locked(size_t list_no, size_t new_size)
-
size_t allocate_slot(size_t capacity)
-
void free_slot(size_t offset, size_t capacity)
-
void set_all_lists_sizes(const size_t *sizes)
override all list sizes and make a packed storage
-
OnDiskInvertedLists()
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const¶
release codes returned by get_codes (default implementation is nop
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const¶
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const¶
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code)¶
add one entry to an inverted list
-
virtual void reset()¶
-
void merge_from(InvertedLists *oivf, size_t add_id)¶
move all entries from oivf (empty on output)
-
double imbalance_factor() const¶
1= perfectly balanced, >1: imbalanced
-
void print_stats() const¶
display some stats about the inverted lists
-
size_t compute_ntotal() const¶
sum up list sizes
Public Members
-
std::vector<List> lists
-
std::list<Slot> slots
-
std::string filename
-
size_t totsize
-
uint8_t *ptr
-
bool read_only
-
LockLevels *locks
-
OngoingPrefetch *pf
-
int prefetch_nthread
-
size_t nlist¶
number of possible key values
-
size_t code_size¶
code size per vector in bytes
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)¶
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
struct Slot
Public Functions
-
Slot(size_t offset, size_t capacity)
-
Slot()
Public Members
-
size_t offset
-
size_t capacity
-
Slot(size_t offset, size_t capacity)
-
struct OnDiskInvertedListsIOHook : public faiss::InvertedListsIOHook
Public Functions
-
OnDiskInvertedListsIOHook()
-
virtual void write(const InvertedLists *ils, IOWriter *f) const override
write the index to the IOWriter (including the fourcc)
-
virtual InvertedLists *read(IOReader *f, int io_flags) const override
called when the fourcc matches this class’s fourcc
-
virtual InvertedLists *read_ArrayInvertedLists(IOReader *f, int io_flags, size_t nlist, size_t code_size, const std::vector<size_t> &sizes) const override
read from a ArrayInvertedLists into this invertedlist type. For this to work, the callback has to be enabled and the io_flag has to be set to IO_FLAG_SKIP_IVF_DATA | (16 upper bits of the fourcc)
(default implementation fails)
Public Members
-
const std::string key¶
string version of the fourcc
-
const std::string classname¶
typeid.name
Public Static Functions
-
static void add_callback(InvertedListsIOHook*)¶
-
static void print_callbacks()¶
-
static InvertedListsIOHook *lookup(int h)¶
-
static InvertedListsIOHook *lookup_classname(const std::string &classname)¶
-
OnDiskInvertedListsIOHook()
-
struct OnDiskOneList
Public Functions
-
OnDiskOneList()
Public Members
-
size_t size
-
size_t capacity
-
size_t offset
-
OnDiskOneList()
-
struct OneRecallAtRCriterion : public faiss::AutoTuneCriterion
-
Public Functions
-
virtual double evaluate(const float *D, const idx_t *I) const override
Evaluate the criterion.
- Parameters:
D – size nq * nnn
I – size nq * nnn
- Returns:
the criterion, between 0 and 1. Larger is better.
-
inline ~OneRecallAtRCriterion() override
-
void set_groundtruth(int gt_nnn, const float *gt_D_in, const idx_t *gt_I_in)
Intitializes the gt_D and gt_I vectors. Must be called before evaluating
- Parameters:
gt_D_in – size nq * gt_nnn
gt_I_in – size nq * gt_nnn
Public Members
-
idx_t R
-
idx_t nq
nb of queries this criterion is evaluated on
-
idx_t nnn
nb of NNs that the query should request
-
idx_t gt_nnn
nb of GT NNs required to evaluate criterion
-
std::vector<float> gt_D
Ground-truth distances (size nq * gt_nnn)
-
std::vector<idx_t> gt_I
Ground-truth indexes (size nq * gt_nnn)
-
virtual double evaluate(const float *D, const idx_t *I) const override
-
struct OperatingPoint
- #include <AutoTune.h>
Maintains a list of experimental results. Each operating point is a (perf, t, key) triplet, where higher perf and lower t is better. The key field is an arbitrary identifier for the operating point.
Includes primitives to extract the Pareto-optimal operating points in the (perf, t) space.
Public Members
-
double perf
performance measure (output of a Criterion)
-
double t
corresponding execution time (ms)
-
std::string key
key that identifies this op pt
-
int64_t cno
integer identifer
-
double perf
-
struct OperatingPoints
Public Functions
-
OperatingPoints()
-
int merge_with(const OperatingPoints &other, const std::string &prefix = "")
add operating points from other to this, with a prefix to the keys
-
void clear()
-
bool add(double perf, double t, const std::string &key, size_t cno = 0)
add a performance measure. Return whether it is an optimal point
-
double t_for_perf(double perf) const
get time required to obtain a given performance measure
-
void display(bool only_optimal = true) const
easy-to-read output
-
void all_to_gnuplot(const char *fname) const
output to a format easy to digest by gnuplot
-
void optimal_to_gnuplot(const char *fname) const
Public Members
-
std::vector<OperatingPoint> all_pts
all operating points
-
std::vector<OperatingPoint> optimal_pts
optimal operating points, sorted by perf
-
OperatingPoints()
-
struct OPQMatrix : public faiss::LinearTransform
- #include <VectorTransform.h>
Applies a rotation to align the dimensions with a PQ to minimize the reconstruction error. Can be used before an IndexPQ or an IndexIVFPQ. The method is the non-parametric version described in:
“Optimized Product Quantization for Approximate Nearest Neighbor Search” Tiezheng Ge, Kaiming He, Qifa Ke, Jian Sun, CVPR’13
Public Functions
-
explicit OPQMatrix(int d = 0, int M = 1, int d2 = -1)
if d2 != -1, output vectors of this dimension
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors. Does nothing by default.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
void transform_transpose(idx_t n, const float *y, float *x) const
compute x = A^T * (x - b) is reverse transform if A has orthonormal lines
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
works only if is_orthonormal
-
void set_is_orthonormal()
compute A^T * A to set the is_orthonormal flag
-
void print_if_verbose(const char *name, const std::vector<double> &mat, int n, int d) const
-
float *apply(idx_t n, const float *x) const
apply the random rotation, return new allocated matrix
- Parameters:
x – size n * d_in
- Returns:
size n * d_out
Public Members
-
int M
nb of subquantizers
-
int niter
Number of outer training iterations.
-
int niter_pq
Number of training iterations for the PQ.
-
int niter_pq_0
same, for the first outer iteration
-
size_t max_train_points
if there are too many training points, resample
-
bool verbose
-
ProductQuantizer *pq
if non-NULL, use this product quantizer for training should be constructed with (d_out, M, _)
-
bool have_bias
-
bool is_orthonormal
! whether to use the bias term
check if matrix A is orthonormal (enables reverse_transform)
-
std::vector<float> A
Transformation matrix, size d_out * d_in.
-
std::vector<float> b
bias vector, size d_out
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit OPQMatrix(int d = 0, int M = 1, int d2 = -1)
-
struct ParameterRange
- #include <AutoTune.h>
possible values of a parameter, sorted from least to most expensive/accurate
Public Members
-
std::string name
-
std::vector<double> values
-
std::string name
-
struct ParameterSpace
- #include <AutoTune.h>
Uses a-priori knowledge on the Faiss indexes to extract tunable parameters.
Subclassed by faiss::gpu::GpuParameterSpace
Public Functions
-
ParameterSpace()
-
size_t n_combinations() const
nb of combinations, = product of values sizes
-
bool combination_ge(size_t c1, size_t c2) const
returns whether combinations c1 >= c2 in the tuple sense
-
std::string combination_name(size_t cno) const
get string representation of the combination
-
void display() const
print a description on stdout
-
ParameterRange &add_range(const std::string &name)
add a new parameter (or return it if it exists)
-
virtual void initialize(const Index *index)
initialize with reasonable parameters for the index
-
void set_index_parameters(Index *index, size_t cno) const
set a combination of parameters on an index
-
void set_index_parameters(Index *index, const char *param_string) const
set a combination of parameters described by a string
-
virtual void set_index_parameter(Index *index, const std::string &name, double val) const
set one of the parameters, returns whether setting was successful
-
void update_bounds(size_t cno, const OperatingPoint &op, double *upper_bound_perf, double *lower_bound_t) const
find an upper bound on the performance and a lower bound on t for configuration cno given another operating point op
-
void explore(Index *index, size_t nq, const float *xq, const AutoTuneCriterion &crit, OperatingPoints *ops) const
explore operating points
- Parameters:
index – index to run on
xq – query vectors (size nq * index.d)
crit – selection criterion
ops – resulting operating points
-
inline virtual ~ParameterSpace()
Public Members
-
std::vector<ParameterRange> parameter_ranges
all tunable parameters
-
int verbose
verbosity during exploration
-
int n_experiments
nb of experiments during optimization (0 = try all combinations)
-
size_t batchsize
maximum number of queries to submit at a time.
-
bool thread_over_batches
use multithreading over batches (useful to benchmark independent single-searches)
-
double min_test_duration
run tests several times until they reach at least this duration (to avoid jittering in MT mode)
-
ParameterSpace()
-
struct PartitionStats
Public Functions
-
inline PartitionStats()
-
void reset()
Public Members
-
uint64_t bissect_cycles
-
uint64_t compress_cycles
-
inline PartitionStats()
-
struct PCAMatrix : public faiss::LinearTransform
- #include <VectorTransform.h>
Applies a principal component analysis on a set of vectors, with optionally whitening and random rotation.
Public Functions
-
explicit PCAMatrix(int d_in = 0, int d_out = 0, float eigen_power = 0, bool random_rotation = false)
-
virtual void train(idx_t n, const float *x) override
train on n vectors. If n < d_in then the eigenvector matrix will be completed with 0s
-
void copy_from(const PCAMatrix &other)
copy pre-trained PCA matrix
-
void prepare_Ab()
called after mean, PCAMat and eigenvalues are computed
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
void transform_transpose(idx_t n, const float *y, float *x) const
compute x = A^T * (x - b) is reverse transform if A has orthonormal lines
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
works only if is_orthonormal
-
void set_is_orthonormal()
compute A^T * A to set the is_orthonormal flag
-
void print_if_verbose(const char *name, const std::vector<double> &mat, int n, int d) const
-
float *apply(idx_t n, const float *x) const
apply the random rotation, return new allocated matrix
- Parameters:
x – size n * d_in
- Returns:
size n * d_out
Public Members
-
float eigen_power
after transformation the components are multiplied by eigenvalues^eigen_power
=0: no whitening =-0.5: full whitening
-
float epsilon
value added to eigenvalues to avoid division by 0 when whitening
-
bool random_rotation
random rotation after PCA
-
size_t max_points_per_d
ratio between # training vectors and dimension
-
int balanced_bins
try to distribute output eigenvectors in this many bins
-
std::vector<float> mean
Mean, size d_in.
-
std::vector<float> eigenvalues
eigenvalues of covariance matrix (= squared singular values)
-
std::vector<float> PCAMat
PCA matrix, size d_in * d_in.
-
bool have_bias
-
bool is_orthonormal
! whether to use the bias term
check if matrix A is orthonormal (enables reverse_transform)
-
std::vector<float> A
Transformation matrix, size d_out * d_in.
-
std::vector<float> b
bias vector, size d_out
-
bool verbose
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
explicit PCAMatrix(int d_in = 0, int d_out = 0, float eigen_power = 0, bool random_rotation = false)
-
struct ProgressiveDimClustering : public faiss::ProgressiveDimClusteringParameters
- #include <Clustering.h>
K-means clustering with progressive dimensions used
The clustering first happens in dim 1, then with exponentially increasing dimension until d (I steps). This is typically applied after a PCA transformation (optional). Reference:
“Improved Residual Vector Quantization for High-dimensional Approximate
Nearest Neighbor Search”
Shicong Liu, Hongtao Lu, Junru Shao, AAAI’15
https://arxiv.org/abs/1509.05195
Public Functions
-
ProgressiveDimClustering(int d, int k)
-
ProgressiveDimClustering(int d, int k, const ProgressiveDimClusteringParameters &cp)
-
void train(idx_t n, const float *x, ProgressiveDimIndexFactory &factory)
-
inline virtual ~ProgressiveDimClustering()
Public Members
-
size_t d
dimension of the vectors
-
size_t k
nb of centroids
-
std::vector<float> centroids
centroids (k * d)
-
std::vector<ClusteringIterationStats> iteration_stats
stats at every iteration of clustering
-
int progressive_dim_steps
number of incremental steps
-
bool apply_pca
apply PCA on input
-
int niter
clustering iterations
-
int nredo
redo clustering this many times and keep best
-
bool verbose
-
bool spherical
do we want normalized centroids?
-
bool int_centroids
round centroids coordinates to integer
-
bool update_index
re-train index after each iteration?
-
bool frozen_centroids
use the centroids provided as input and do not change them during iterations
-
int min_points_per_centroid
otherwise you get a warning
-
int max_points_per_centroid
to limit size of dataset
-
int seed
seed for the random number generator
-
size_t decode_block_size
how many vectors at a time to decode
-
ProgressiveDimClustering(int d, int k)
-
struct ProgressiveDimClusteringParameters : public faiss::ClusteringParameters
Subclassed by faiss::ProgressiveDimClustering
Public Functions
-
ProgressiveDimClusteringParameters()
Public Members
-
int progressive_dim_steps
number of incremental steps
-
bool apply_pca
apply PCA on input
-
int niter
clustering iterations
-
int nredo
redo clustering this many times and keep best
-
bool verbose
-
bool spherical
do we want normalized centroids?
-
bool int_centroids
round centroids coordinates to integer
-
bool update_index
re-train index after each iteration?
-
bool frozen_centroids
use the centroids provided as input and do not change them during iterations
-
int min_points_per_centroid
otherwise you get a warning
-
int max_points_per_centroid
to limit size of dataset
-
int seed
seed for the random number generator
-
size_t decode_block_size
how many vectors at a time to decode
-
ProgressiveDimClusteringParameters()
-
struct ProgressiveDimIndexFactory
- #include <Clustering.h>
generates an index suitable for clustering when called
Subclassed by faiss::gpu::GpuProgressiveDimIndexFactory
Public Functions
-
virtual Index *operator()(int dim)
ownership transferred to caller
-
inline virtual ~ProgressiveDimIndexFactory()
-
virtual Index *operator()(int dim)
-
struct RandomGenerator
- #include <random.h>
random generator that can be used in multithreaded contexts
Public Functions
-
int rand_int()
random positive integer
-
int64_t rand_int64()
random int64_t
-
int rand_int(int max)
generate random integer between 0 and max-1
-
float rand_float()
between 0 and 1
-
double rand_double()
-
explicit RandomGenerator(int64_t seed = 1234)
Public Members
-
std::mt19937 mt
-
int rand_int()
-
struct RandomRotationMatrix : public faiss::LinearTransform
- #include <VectorTransform.h>
Randomly rotate a set of vectors.
Public Functions
-
inline RandomRotationMatrix(int d_in, int d_out)
both d_in > d_out and d_out < d_in are supported
-
void init(int seed)
must be called before the transform is used
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors. Does nothing by default.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
inline RandomRotationMatrix()
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
void transform_transpose(idx_t n, const float *y, float *x) const
compute x = A^T * (x - b) is reverse transform if A has orthonormal lines
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
works only if is_orthonormal
-
void set_is_orthonormal()
compute A^T * A to set the is_orthonormal flag
-
void print_if_verbose(const char *name, const std::vector<double> &mat, int n, int d) const
-
float *apply(idx_t n, const float *x) const
apply the random rotation, return new allocated matrix
- Parameters:
x – size n * d_in
- Returns:
size n * d_out
Public Members
-
bool have_bias
-
bool is_orthonormal
! whether to use the bias term
check if matrix A is orthonormal (enables reverse_transform)
-
std::vector<float> A
Transformation matrix, size d_out * d_in.
-
std::vector<float> b
bias vector, size d_out
-
bool verbose
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
inline RandomRotationMatrix(int d_in, int d_out)
-
struct ReadOnlyInvertedLists : public faiss::InvertedLists
- #include <InvertedLists.h>
invlists that fail for all write functions
Subclassed by faiss::HStackInvertedLists, faiss::MaskedInvertedLists, faiss::SliceInvertedLists, faiss::StopWordsInvertedLists, faiss::VStackInvertedLists
Public Functions
-
inline ReadOnlyInvertedLists(size_t nlist, size_t code_size)
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void resize(size_t list_no, size_t new_size) override
-
virtual size_t list_size(size_t list_no) const = 0
get the size of a list
-
virtual const uint8_t *get_codes(size_t list_no) const = 0
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const = 0
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const
release codes returned by get_codes (default implementation is nop
-
virtual void release_ids(size_t list_no, const idx_t *ids) const
release ids returned by get_ids
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code)
add one entry to an inverted list
-
virtual void update_entry(size_t list_no, size_t offset, idx_t id, const uint8_t *code)
-
virtual void reset()
-
void merge_from(InvertedLists *oivf, size_t add_id)
move all entries from oivf (empty on output)
-
double imbalance_factor() const
1= perfectly balanced, >1: imbalanced
-
void print_stats() const
display some stats about the inverted lists
-
size_t compute_ntotal() const
sum up list sizes
Public Members
-
size_t nlist
number of possible key values
-
size_t code_size
code size per vector in bytes
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
inline ReadOnlyInvertedLists(size_t nlist, size_t code_size)
-
struct ReconstructFromNeighbors
-
Public Functions
-
explicit ReconstructFromNeighbors(const IndexHNSW &index, size_t k = 256, size_t nsq = 1)
-
void add_codes(size_t n, const float *x)
codes must be added in the correct order and the IndexHNSW must be populated and sorted
-
size_t compute_distances(size_t n, const idx_t *shortlist, const float *query, float *distances) const
-
void estimate_code(const float *x, storage_idx_t i, uint8_t *code) const
called by add_codes
-
void reconstruct(storage_idx_t i, float *x, float *tmp) const
called by compute_distances
-
void reconstruct_n(storage_idx_t n0, storage_idx_t ni, float *x) const
-
void get_neighbor_table(storage_idx_t i, float *out) const
get the M+1 -by-d table for neighbor coordinates for vector i
Public Members
-
const IndexHNSW &index
-
size_t M
-
size_t k
-
size_t nsq
-
size_t code_size
-
int k_reorder
-
std::vector<float> codebook
-
std::vector<uint8_t> codes
-
size_t ntotal
-
size_t d
-
size_t dsub
-
explicit ReconstructFromNeighbors(const IndexHNSW &index, size_t k = 256, size_t nsq = 1)
-
struct RemapDimensionsTransform : public faiss::VectorTransform
- #include <VectorTransform.h>
remap dimensions for intput vectors, possibly inserting 0s strictly speaking this is also a linear transform but we don’t want to compute it with matrix multiplies
Public Functions
-
RemapDimensionsTransform(int d_in, int d_out, const int *map)
-
RemapDimensionsTransform(int d_in, int d_out, bool uniform = true)
remap input to output, skipping or inserting dimensions as needed if uniform: distribute dimensions uniformly otherwise just take the d_out first ones.
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const override
same as apply, but result is pre-allocated
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const override
reverse transform correct only when the mapping is a permutation
-
inline RemapDimensionsTransform()
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors. Does nothing by default.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
float *apply(idx_t n, const float *x) const
apply the random rotation, return new allocated matrix
- Parameters:
x – size n * d_in
- Returns:
size n * d_out
Public Members
-
std::vector<int> map
map from output dimension to input, size d_out -1 -> set output to 0
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
RemapDimensionsTransform(int d_in, int d_out, const int *map)
-
struct ResidualCoarseQuantizer : public faiss::AdditiveCoarseQuantizer
- #include <IndexAdditiveQuantizer.h>
The ResidualCoarseQuantizer is a bit specialized compared to the default AdditiveCoarseQuantizer because it can use a beam search at search time (slow but may be useful for very large vocabularies)
Public Types
-
using idx_t = int64_t¶
all indices are this type
-
using component_t = float¶
-
using distance_t = float¶
Public Functions
-
void set_beam_factor(float new_beam_factor)
computes centroid norms if required
-
ResidualCoarseQuantizer(int d, size_t M, size_t nbits, MetricType metric = METRIC_L2)
Constructor.
- Parameters:
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
d – dimensionality of the input vectors
M – number of subquantizers
nbits – number of bit per subvector index
-
ResidualCoarseQuantizer(int d, const std::vector<size_t> &nbits, MetricType metric = METRIC_L2)
-
virtual void search(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels) const override
query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
ResidualCoarseQuantizer()
-
virtual void add(idx_t n, const float *x) override
N/A.
-
virtual void reconstruct(idx_t key, float *recons) const override
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void train(idx_t n, const float *x) override
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void reset() override
N/A.
-
virtual void add_with_ids(idx_t n, const float *x, const idx_t *xids)¶
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const¶
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void assign(idx_t n, const float *x, idx_t *labels, idx_t k = 1) const¶
return the indexes of the k vectors closest to the query x.
This function is identical as search but only return labels of neighbors.
- Parameters:
x – input vectors to search, size n * d
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)¶
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const¶
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const¶
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const¶
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual void compute_residual(const float *x, float *residual, idx_t key) const¶
Computes a residual vector after indexing encoding.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
x – input vector, size d
residual – output residual vector, size d
key – encoded index, as returned by search and assign
-
virtual void compute_residual_n(idx_t n, const float *xs, float *residuals, const idx_t *keys) const¶
Computes a residual vector after indexing encoding (batch form). Equivalent to calling compute_residual for each vector.
The residual vector is the difference between a vector and the reconstruction that can be decoded from its representation in the index. The residual can be used for multiple-stage indexing methods, like IndexIVF’s methods.
- Parameters:
n – number of vectors
xs – input vectors, size (n x d)
residuals – output residual vectors, size (n x d)
keys – encoded index, as returned by search and assign
-
virtual DistanceComputer *get_distance_computer() const¶
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const¶
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const¶
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const¶
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
ResidualQuantizer rq
The residual quantizer used to encode the vectors.
-
float beam_factor
factor between the beam size and the search k if negative, use exact search-to-centroid
-
AdditiveQuantizer *aq
-
std::vector<float> centroid_norms
norms of centroids, useful for knn-search
-
int d¶
vector dimension
-
bool verbose¶
verbosity level
-
MetricType metric_type¶
type of metric this index uses for search
-
float metric_arg¶
argument of the metric type
-
using idx_t = int64_t¶
-
struct simd16uint16 : public faiss::simd256bit
- #include <simdlib_avx2.h>
vector of 16 elements in uint16
Public Functions
-
inline simd16uint16()
-
inline explicit simd16uint16(__m256i i)
-
inline explicit simd16uint16(int x)
-
inline explicit simd16uint16(uint16_t x)
-
inline explicit simd16uint16(simd256bit x)
-
inline explicit simd16uint16(const uint16_t *x)
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint16_t x)
-
inline simd16uint16 operator*(const simd16uint16 &other) const
-
inline simd16uint16 operator>>(const int shift) const
-
inline simd16uint16 operator<<(const int shift) const
-
inline simd16uint16 operator+=(simd16uint16 other)
-
inline simd16uint16 operator-=(simd16uint16 other)
-
inline simd16uint16 operator+(simd16uint16 other) const
-
inline simd16uint16 operator-(simd16uint16 other) const
-
inline simd16uint16 operator&(simd256bit other) const
-
inline simd16uint16 operator|(simd256bit other) const
-
inline simd16uint16 operator==(simd256bit other) const
-
inline simd16uint16 operator~() const
-
inline uint16_t get_scalar_0() const
-
inline uint32_t ge_mask(simd16uint16 thresh) const
-
inline uint32_t le_mask(simd16uint16 thresh) const
-
inline uint32_t gt_mask(simd16uint16 thresh) const
-
inline bool all_gt(simd16uint16 thresh) const
-
inline uint16_t operator[](int i) const
-
inline void accu_min(simd16uint16 incoming)
-
inline void accu_max(simd16uint16 incoming)
-
inline simd16uint16()
-
inline explicit simd16uint16(int x)
-
inline explicit simd16uint16(uint16_t x)
-
inline explicit simd16uint16(const simd256bit &x)¶
-
inline explicit simd16uint16(const uint16_t *x)
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint16_t x)
-
inline simd16uint16 operator*(const simd16uint16 &other) const
-
inline simd16uint16 operator>>(const int shift) const
-
inline simd16uint16 operator<<(const int shift) const
-
inline simd16uint16 operator+=(const simd16uint16 &other)¶
-
inline simd16uint16 operator-=(const simd16uint16 &other)¶
-
inline simd16uint16 operator+(const simd16uint16 &other) const¶
-
inline simd16uint16 operator-(const simd16uint16 &other) const¶
-
inline simd16uint16 operator&(const simd256bit &other) const¶
-
inline simd16uint16 operator|(const simd256bit &other) const¶
-
inline simd16uint16 operator==(const simd16uint16 &other) const¶
-
inline simd16uint16 operator~() const
-
inline uint16_t get_scalar_0() const
-
inline uint32_t ge_mask(const simd16uint16 &thresh) const¶
-
inline uint32_t le_mask(const simd16uint16 &thresh) const¶
-
inline uint32_t gt_mask(const simd16uint16 &thresh) const¶
-
inline bool all_gt(const simd16uint16 &thresh) const¶
-
inline uint16_t operator[](int i) const
-
inline void accu_min(const simd16uint16 &incoming)¶
-
inline void accu_max(const simd16uint16 &incoming)¶
-
simd16uint16() = default
-
inline explicit simd16uint16(int x)
-
inline explicit simd16uint16(uint16_t x)
-
inline explicit simd16uint16(const uint16x8x2_t &v)¶
-
template<typename T, typename std::enable_if<detail::simdlib::is_simd256bit<T>::value, std::nullptr_t>::type = nullptr>
inline explicit simd16uint16(const T &x)¶
-
inline explicit simd16uint16(const uint16_t *x)
-
inline void clear()¶
-
inline void storeu(uint16_t *ptr) const¶
-
inline void loadu(const uint16_t *ptr)¶
-
inline void store(uint16_t *ptr) const¶
-
inline void bin(char bits[257]) const¶
-
inline std::string bin() const¶
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint16_t x)
-
inline simd16uint16 operator*(const simd16uint16 &other) const
-
inline simd16uint16 operator>>(const int shift) const
-
inline simd16uint16 operator<<(const int shift) const
-
inline simd16uint16 operator+=(const simd16uint16 &other)
-
inline simd16uint16 operator-=(const simd16uint16 &other)
-
inline simd16uint16 operator+(const simd16uint16 &other) const
-
inline simd16uint16 operator-(const simd16uint16 &other) const
-
template<typename T, typename std::enable_if<detail::simdlib::is_simd256bit<T>::value, std::nullptr_t>::type = nullptr>
inline simd16uint16 operator&(const T &other) const¶
-
template<typename T, typename std::enable_if<detail::simdlib::is_simd256bit<T>::value, std::nullptr_t>::type = nullptr>
inline simd16uint16 operator|(const T &other) const¶
-
inline simd16uint16 operator==(const simd16uint16 &other) const
-
inline simd16uint16 operator~() const
-
inline uint16_t get_scalar_0() const
-
inline uint32_t ge_mask(const simd16uint16 &thresh) const
-
inline uint32_t le_mask(const simd16uint16 &thresh) const
-
inline uint32_t gt_mask(const simd16uint16 &thresh) const
-
inline bool all_gt(const simd16uint16 &thresh) const
-
inline uint16_t operator[](int i) const
-
inline void accu_min(const simd16uint16 &incoming)
-
inline void accu_max(const simd16uint16 &incoming)
-
inline void storeu(void *ptr) const
-
inline void loadu(const void *ptr)
-
inline void store(void *ptr) const
Public Members
-
uint16x8x2_t data¶
-
__m256i i
-
__m256 f
-
union faiss::simd256bit::[anonymous] [anonymous]
-
uint8_t u8[32]¶
-
uint16_t u16[16]¶
-
uint32_t u32[8]¶
-
float f32[8]¶
-
union faiss::simd256bit::[anonymous] [anonymous]¶
Public Static Functions
-
template<typename F>
static inline simd16uint16 unary_func(const simd16uint16 &a, F &&f)¶
-
template<typename F>
static inline simd16uint16 binary_func(const simd16uint16 &a, const simd16uint16 &b, F &&f)¶
-
inline simd16uint16()
-
struct simd256bit
- #include <simdlib_avx2.h>
256-bit representation without interpretation as a vector
Simple wrapper around the AVX 256-bit registers
The objective is to separate the different interpretations of the same registers (as a vector of uint8, uint16 or uint32), to provide printing functions, and to give more readable names to the AVX intrinsics. It does not pretend to be exhausitve, functions are added as needed.
Subclassed by faiss::simd16uint16, faiss::simd32uint8, faiss::simd8float32, faiss::simd8uint32
Public Functions
-
inline simd256bit()
-
inline explicit simd256bit(__m256i i)
-
inline explicit simd256bit(__m256 f)
-
inline explicit simd256bit(const void *x)
-
inline void clear()
-
inline void storeu(void *ptr) const
-
inline void loadu(const void *ptr)
-
inline void store(void *ptr) const
-
inline void bin(char bits[257]) const
-
inline std::string bin() const
-
inline simd256bit()
-
inline explicit simd256bit(const void *x)
-
inline void clear()
-
inline void storeu(void *ptr) const
-
inline void loadu(const void *ptr)
-
inline void store(void *ptr) const
-
inline void bin(char bits[257]) const
-
inline std::string bin() const
Public Members
-
__m256i i
-
__m256 f
-
union faiss::simd256bit::[anonymous] [anonymous]
-
uint8_t u8[32]¶
-
uint16_t u16[16]¶
-
uint32_t u32[8]¶
-
float f32[8]¶
-
union faiss::simd256bit::[anonymous] [anonymous]¶
-
inline simd256bit()
-
struct simd32uint8 : public faiss::simd256bit
Public Functions
-
inline simd32uint8()
-
inline explicit simd32uint8(__m256i i)
-
inline explicit simd32uint8(int x)
-
inline explicit simd32uint8(uint8_t x)
-
inline explicit simd32uint8(simd256bit x)
-
inline explicit simd32uint8(const uint8_t *x)
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint8_t x)
-
inline simd32uint8 operator&(simd256bit other) const
-
inline simd32uint8 operator+(simd32uint8 other) const
-
inline simd32uint8 lookup_2_lanes(simd32uint8 idx) const
-
inline simd16uint16 lane0_as_uint16() const
-
inline simd16uint16 lane1_as_uint16() const
-
inline simd32uint8 operator+=(simd32uint8 other)
-
inline uint8_t operator[](int i) const
-
inline simd32uint8()
-
inline explicit simd32uint8(int x)
-
inline explicit simd32uint8(uint8_t x)
-
inline explicit simd32uint8(const simd256bit &x)¶
-
inline explicit simd32uint8(const uint8_t *x)
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint8_t x)
-
inline simd32uint8 operator&(const simd256bit &other) const¶
-
inline simd32uint8 operator+(const simd32uint8 &other) const¶
-
inline simd32uint8 lookup_2_lanes(const simd32uint8 &idx) const¶
-
inline simd32uint8 operator+=(const simd32uint8 &other)¶
-
inline uint8_t operator[](int i) const
-
simd32uint8() = default
-
inline explicit simd32uint8(int x)
-
inline explicit simd32uint8(uint8_t x)
-
inline explicit simd32uint8(const uint8x16x2_t &v)¶
-
template<typename T, typename std::enable_if<detail::simdlib::is_simd256bit<T>::value, std::nullptr_t>::type = nullptr>
inline explicit simd32uint8(const T &x)¶
-
inline explicit simd32uint8(const uint8_t *x)
-
inline void clear()¶
-
inline void storeu(uint8_t *ptr) const¶
-
inline void loadu(const uint8_t *ptr)¶
-
inline void store(uint8_t *ptr) const¶
-
inline void bin(char bits[257]) const¶
-
inline std::string bin() const¶
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint8_t x)
-
template<typename T, typename std::enable_if<detail::simdlib::is_simd256bit<T>::value, std::nullptr_t>::type = nullptr>
inline simd32uint8 operator&(const T &other) const¶
-
inline simd32uint8 operator+(const simd32uint8 &other) const
-
inline simd32uint8 lookup_2_lanes(const simd32uint8 &idx) const
-
inline simd32uint8 operator+=(const simd32uint8 &other)
-
inline uint8_t operator[](int i) const
-
inline void storeu(void *ptr) const
-
inline void loadu(const void *ptr)
-
inline void store(void *ptr) const
Public Members
-
uint8x16x2_t data¶
-
__m256i i
-
__m256 f
-
union faiss::simd256bit::[anonymous] [anonymous]
-
uint8_t u8[32]¶
-
uint16_t u16[16]¶
-
uint32_t u32[8]¶
-
float f32[8]¶
-
union faiss::simd256bit::[anonymous] [anonymous]¶
Public Static Functions
-
template<typename F>
static inline simd32uint8 binary_func(const simd32uint8 &a, const simd32uint8 &b, F &&f)¶
-
inline simd32uint8()
-
struct simd8float32 : public faiss::simd256bit
Public Functions
-
inline simd8float32()
-
inline explicit simd8float32(simd256bit x)
-
inline explicit simd8float32(__m256 x)
-
inline explicit simd8float32(float x)
-
inline explicit simd8float32(const float *x)
-
inline simd8float32 operator*(simd8float32 other) const
-
inline simd8float32 operator+(simd8float32 other) const
-
inline simd8float32 operator-(simd8float32 other) const
-
inline std::string tostring() const
-
inline simd8float32()
-
inline explicit simd8float32(const simd256bit &x)¶
-
inline explicit simd8float32(float x)
-
inline explicit simd8float32(const float *x)
-
inline void set1(float x)¶
-
inline simd8float32 operator*(const simd8float32 &other) const¶
-
inline simd8float32 operator+(const simd8float32 &other) const¶
-
inline simd8float32 operator-(const simd8float32 &other) const¶
-
inline std::string tostring() const
-
simd8float32() = default
-
inline explicit simd8float32(float x)
-
inline explicit simd8float32(const float32x4x2_t &v)¶
-
template<typename T, typename std::enable_if<detail::simdlib::is_simd256bit<T>::value, std::nullptr_t>::type = nullptr>
inline explicit simd8float32(const T &x)¶
-
inline explicit simd8float32(const float *x)
-
inline void clear()¶
-
inline void storeu(float *ptr) const¶
-
inline void loadu(const float *ptr)¶
-
inline void store(float *ptr) const¶
-
inline void bin(char bits[257]) const¶
-
inline std::string bin() const¶
-
inline simd8float32 operator*(const simd8float32 &other) const
-
inline simd8float32 operator+(const simd8float32 &other) const
-
inline simd8float32 operator-(const simd8float32 &other) const
-
inline std::string tostring() const
-
inline void storeu(void *ptr) const
-
inline void loadu(const void *ptr)
-
inline void store(void *ptr) const
Public Members
-
float32x4x2_t data¶
-
__m256i i
-
__m256 f
-
union faiss::simd256bit::[anonymous] [anonymous]
-
uint8_t u8[32]¶
-
uint16_t u16[16]¶
-
uint32_t u32[8]¶
-
float f32[8]¶
-
union faiss::simd256bit::[anonymous] [anonymous]¶
Public Static Functions
-
template<typename F>
static inline simd8float32 binary_func(const simd8float32 &a, const simd8float32 &b, F &&f)¶
-
inline simd8float32()
-
struct simd8uint32 : public faiss::simd256bit
- #include <simdlib_avx2.h>
vector of 8 unsigned 32-bit integers
Public Functions
-
inline simd8uint32()
-
inline explicit simd8uint32(__m256i i)
-
inline explicit simd8uint32(uint32_t x)
-
inline explicit simd8uint32(simd256bit x)
-
inline explicit simd8uint32(const uint8_t *x)
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint32_t x)
-
inline simd8uint32()
-
inline explicit simd8uint32(uint32_t x)
-
inline explicit simd8uint32(const simd256bit &x)¶
-
inline explicit simd8uint32(const uint8_t *x)
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint32_t x)
-
simd8uint32() = default
-
inline explicit simd8uint32(uint32_t x)
-
inline explicit simd8uint32(const uint32x4x2_t &v)¶
-
template<typename T, typename std::enable_if<detail::simdlib::is_simd256bit<T>::value, std::nullptr_t>::type = nullptr>
inline explicit simd8uint32(const T &x)¶
-
inline explicit simd8uint32(const uint8_t *x)
-
inline void clear()¶
-
inline void storeu(uint32_t *ptr) const¶
-
inline void loadu(const uint32_t *ptr)¶
-
inline void store(uint32_t *ptr) const¶
-
inline void bin(char bits[257]) const¶
-
inline std::string bin() const¶
-
inline std::string elements_to_string(const char *fmt) const
-
inline std::string hex() const
-
inline std::string dec() const
-
inline void set1(uint32_t x)
-
inline void storeu(void *ptr) const
-
inline void loadu(const void *ptr)
-
inline void store(void *ptr) const
Public Members
-
uint32x4x2_t data¶
-
__m256i i
-
__m256 f
-
union faiss::simd256bit::[anonymous] [anonymous]
-
uint8_t u8[32]¶
-
uint16_t u16[16]¶
-
uint32_t u32[8]¶
-
float f32[8]¶
-
union faiss::simd256bit::[anonymous] [anonymous]¶
-
inline simd8uint32()
-
struct SliceInvertedLists : public faiss::ReadOnlyInvertedLists
- #include <InvertedLists.h>
vertical slice of indexes in another InvertedLists
Public Functions
-
SliceInvertedLists(const InvertedLists *il, idx_t i0, idx_t i1)
-
virtual size_t list_size(size_t list_no) const override
get the size of a list
-
virtual const uint8_t *get_codes(size_t list_no) const override
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const override
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const override
release codes returned by get_codes (default implementation is nop
-
void release_ids(size_t list_no, const idx_t *ids) const override
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const override
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const override
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
void prefetch_lists(const idx_t *list_nos, int nlist) const override
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void resize(size_t list_no, size_t new_size) override
-
virtual void release_ids(size_t list_no, const idx_t *ids) const
release ids returned by get_ids
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code)
add one entry to an inverted list
-
virtual void update_entry(size_t list_no, size_t offset, idx_t id, const uint8_t *code)
-
virtual void reset()
-
void merge_from(InvertedLists *oivf, size_t add_id)
move all entries from oivf (empty on output)
-
double imbalance_factor() const
1= perfectly balanced, >1: imbalanced
-
void print_stats() const
display some stats about the inverted lists
-
size_t compute_ntotal() const
sum up list sizes
Public Members
-
const InvertedLists *il
-
idx_t i0
-
idx_t i1
-
size_t nlist
number of possible key values
-
size_t code_size
code size per vector in bytes
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
SliceInvertedLists(const InvertedLists *il, idx_t i0, idx_t i1)
-
struct StopWordsInvertedLists : public faiss::ReadOnlyInvertedLists
- #include <InvertedLists.h>
if the inverted list in il is smaller than maxsize then return it, otherwise return an empty invlist
Public Functions
-
StopWordsInvertedLists(const InvertedLists *il, size_t maxsize)
-
virtual size_t list_size(size_t list_no) const override
get the size of a list
-
virtual const uint8_t *get_codes(size_t list_no) const override
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const override
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const override
release codes returned by get_codes (default implementation is nop
-
virtual void release_ids(size_t list_no, const idx_t *ids) const override
release ids returned by get_ids
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const override
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const override
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const override
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void resize(size_t list_no, size_t new_size) override
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code)
add one entry to an inverted list
-
virtual void update_entry(size_t list_no, size_t offset, idx_t id, const uint8_t *code)
-
virtual void reset()
-
void merge_from(InvertedLists *oivf, size_t add_id)
move all entries from oivf (empty on output)
-
double imbalance_factor() const
1= perfectly balanced, >1: imbalanced
-
void print_stats() const
display some stats about the inverted lists
-
size_t compute_ntotal() const
sum up list sizes
Public Members
-
const InvertedLists *il0
-
size_t maxsize
-
size_t nlist
number of possible key values
-
size_t code_size
code size per vector in bytes
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
StopWordsInvertedLists(const InvertedLists *il, size_t maxsize)
-
template<MetricType mt>
struct VectorDistance Public Types
-
using C = typename std::conditional<mt == METRIC_INNER_PRODUCT, CMin<float, int64_t>, CMax<float, int64_t>>::type
Public Functions
-
inline float operator()(const float *x, const float *y) const
-
inline float operator()(const float *x, const float *y) const
-
inline float operator()(const float *x, const float *y) const
-
inline float operator()(const float *x, const float *y) const
-
inline float operator()(const float *x, const float *y) const
-
inline float operator()(const float *x, const float *y) const
-
inline float operator()(const float *x, const float *y) const
-
inline float operator()(const float *x, const float *y) const
-
inline float operator()(const float *x, const float *y) const
Public Members
-
size_t d
-
float metric_arg
-
using C = typename std::conditional<mt == METRIC_INNER_PRODUCT, CMin<float, int64_t>, CMax<float, int64_t>>::type
-
struct VectorTransform
- #include <VectorTransform.h>
Any transformation applied on a set of vectors
Subclassed by faiss::CenteringTransform, faiss::ITQTransform, faiss::LinearTransform, faiss::NormalizationTransform, faiss::RemapDimensionsTransform
Public Functions
-
inline explicit VectorTransform(int d_in = 0, int d_out = 0)
! output dimension
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors. Does nothing by default.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
float *apply(idx_t n, const float *x) const
apply the random rotation, return new allocated matrix
- Parameters:
x – size n * d_in
- Returns:
size n * d_out
-
virtual void apply_noalloc(idx_t n, const float *x, float *xt) const = 0
same as apply, but result is pre-allocated
-
virtual void reverse_transform(idx_t n, const float *xt, float *x) const
reverse transformation. May not be implemented or may return approximate result
-
inline virtual ~VectorTransform()
Public Members
-
int d_in
-
int d_out
! input dimension
-
bool is_trained
set if the VectorTransform does not require training, or if training is done already
-
inline explicit VectorTransform(int d_in = 0, int d_out = 0)
-
struct VStackInvertedLists : public faiss::ReadOnlyInvertedLists
-
Public Functions
-
VStackInvertedLists(int nil, const InvertedLists **ils)
build InvertedLists by concatenating nil of them
-
virtual size_t list_size(size_t list_no) const override
get the size of a list
-
virtual const uint8_t *get_codes(size_t list_no) const override
get the codes for an inverted list must be released by release_codes
- Returns:
codes size list_size * code_size
-
virtual const idx_t *get_ids(size_t list_no) const override
get the ids for an inverted list must be released by release_ids
- Returns:
ids size list_size
-
virtual void release_codes(size_t list_no, const uint8_t *codes) const override
release codes returned by get_codes (default implementation is nop
-
virtual void release_ids(size_t list_no, const idx_t *ids) const override
release ids returned by get_ids
-
virtual idx_t get_single_id(size_t list_no, size_t offset) const override
- Returns:
a single id in an inverted list
-
virtual const uint8_t *get_single_code(size_t list_no, size_t offset) const override
- Returns:
a single code in an inverted list (should be deallocated with release_codes)
-
virtual void prefetch_lists(const idx_t *list_nos, int nlist) const override
prepare the following lists (default does nothing) a list can be -1 hence the signed long
-
virtual size_t add_entries(size_t list_no, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void update_entries(size_t list_no, size_t offset, size_t n_entry, const idx_t *ids, const uint8_t *code) override
-
virtual void resize(size_t list_no, size_t new_size) override
-
virtual size_t add_entry(size_t list_no, idx_t theid, const uint8_t *code)
add one entry to an inverted list
-
virtual void update_entry(size_t list_no, size_t offset, idx_t id, const uint8_t *code)
-
virtual void reset()
-
void merge_from(InvertedLists *oivf, size_t add_id)
move all entries from oivf (empty on output)
-
double imbalance_factor() const
1= perfectly balanced, >1: imbalanced
-
void print_stats() const
display some stats about the inverted lists
-
size_t compute_ntotal() const
sum up list sizes
Public Members
-
std::vector<const InvertedLists*> ils
-
std::vector<idx_t> cumsz
-
size_t nlist
number of possible key values
-
size_t code_size
code size per vector in bytes
Public Static Attributes
-
static const size_t INVALID_CODE_SIZE = static_cast<size_t>(-1)
used for BlockInvertedLists, where the codes are packed into groups and the individual code size is meaningless
-
VStackInvertedLists(int nil, const InvertedLists **ils)
-
class WorkerThread
Public Functions
-
WorkerThread()
-
~WorkerThread()
Stops and waits for the worker thread to exit, flushing all pending lambdas
-
void stop()
Request that the worker thread stop itself.
-
void waitForThreadExit()
Blocking waits in the current thread for the worker thread to stop
-
std::future<bool> add(std::function<void()> f)
Adds a lambda to run on the worker thread; returns a future that can be used to block on its completion. Future status is
trueif the lambda was run in the worker thread;falseif it was not run, because the worker thread is exiting or has exited.
Private Functions
-
void startThread()
-
void threadMain()
-
void threadLoop()
Private Members
-
std::thread thread_
Thread that all queued lambdas are run on.
-
std::mutex mutex_
Mutex for the queue and exit status.
-
std::condition_variable monitor_
Monitor for the exit status and the queue.
-
bool wantStop_
Whether or not we want the thread to exit.
-
std::deque<std::pair<std::function<void()>, std::promise<bool>>> queue_
Queue of pending lambdas to call.
-
WorkerThread()
-
namespace cppcontrib
-
template<intptr_t DIM, intptr_t COARSE_SIZE, intptr_t FINE_SIZE>
struct Index2LevelDecoder Public Static Functions
- static inline void store (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, float *const __restrict outputStore)
- static inline void accum (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, const float weight, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids0, const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqCoarseCentroids1, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum)
- static inline void store (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, float *const __restrict outputStore)
- static inline void accum (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, const float weight, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids0, const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqCoarseCentroids1, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum)
- static inline void store (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, float *const __restrict outputStore)
- static inline void accum (const float *const __restrict pqCoarseCentroids, const float *const __restrict pqFineCentroids, const uint8_t *const __restrict code, const float weight, float *const __restrict outputAccum)
- static inline void accum (const float *const __restrict pqCoarseCentroids0, const float *const __restrict pqFineCentroids0, const uint8_t *const __restrict code0, const float weight0, const float *const __restrict pqCoarseCentroids1, const float *const __restrict pqFineCentroids1, const uint8_t *const __restrict code1, const float weight1, float *const __restrict outputAccum)
-
template<intptr_t DIM, intptr_t COARSE_SIZE, intptr_t FINE_SIZE>
-
namespace detail
-
namespace simdlib
Functions
-
static inline uint8x16x2_t reinterpret_u8(const uint8x16x2_t &v)
-
static inline uint8x16x2_t reinterpret_u8(const uint16x8x2_t &v)
-
static inline uint8x16x2_t reinterpret_u8(const uint32x4x2_t &v)
-
static inline uint8x16x2_t reinterpret_u8(const float32x4x2_t &v)
-
static inline uint16x8x2_t reinterpret_u16(const uint8x16x2_t &v)
-
static inline uint16x8x2_t reinterpret_u16(const uint16x8x2_t &v)
-
static inline uint16x8x2_t reinterpret_u16(const uint32x4x2_t &v)
-
static inline uint16x8x2_t reinterpret_u16(const float32x4x2_t &v)
-
static inline uint32x4x2_t reinterpret_u32(const uint8x16x2_t &v)
-
static inline uint32x4x2_t reinterpret_u32(const uint16x8x2_t &v)
-
static inline uint32x4x2_t reinterpret_u32(const uint32x4x2_t &v)
-
static inline uint32x4x2_t reinterpret_u32(const float32x4x2_t &v)
-
static inline float32x4x2_t reinterpret_f32(const uint8x16x2_t &v)
-
static inline float32x4x2_t reinterpret_f32(const uint16x8x2_t &v)
-
static inline float32x4x2_t reinterpret_f32(const uint32x4x2_t &v)
-
static inline float32x4x2_t reinterpret_f32(const float32x4x2_t &v)
-
static inline void bin(const char (&bytes)[32], char bits[257])
-
template<typename T, size_t N, typename S>
static inline void bin(const S &simd, char bits[257])
-
template<typename S>
static inline std::string bin(const S &simd)
-
template<typename T, size_t N, typename S>
static inline std::string elements_to_string(const char *fmt, const S &simd)
-
static inline uint16_t vmovmask_u8(const uint8x16_t &v)
-
template<uint16x8_t (*F)(uint16x8_t, uint16x8_t)>
static inline uint32_t cmp_xe32(const uint16x8x2_t &d0, const uint16x8x2_t &d1, const uint16x8x2_t &thr)
-
template<std::uint8_t Shift>
static inline uint16x8_t vshlq(uint16x8_t vec)
-
template<std::uint8_t Shift>
static inline uint16x8_t vshrq(uint16x8_t vec)
-
template<typename T, typename U = decltype(reinterpret_u8(std::declval<T>().data))>
struct is_simd256bit : public std::is_same<decltype(reinterpret_u8(std::declval<T>().data)), uint8x16x2_t>
-
static inline uint8x16x2_t reinterpret_u8(const uint8x16x2_t &v)
-
namespace simdlib
-
namespace gpu
Enums
-
enum class DistanceDataType
Values:
-
enumerator F32
-
enumerator F16
-
enumerator F32
-
enum class IndicesDataType
Values:
-
enumerator I64
-
enumerator I32
-
enumerator I64
-
enum IndicesOptions
How user vector index data is stored on the GPU.
Values:
-
enumerator INDICES_CPU
The user indices are only stored on the CPU; the GPU returns (inverted list, offset) to the CPU which is then translated to the real user index.
-
enumerator INDICES_IVF
The indices are not stored at all, on either the CPU or GPU. Only (inverted list, offset) is returned to the user as the index.
-
enumerator INDICES_32_BIT
Indices are stored as 32 bit integers on the GPU, but returned as 64 bit integers
-
enumerator INDICES_64_BIT
Indices are stored as 64 bit integers on the GPU.
-
enumerator INDICES_CPU
-
enum AllocType
Values:
-
enumerator Other
Unknown allocation type or miscellaneous (not currently categorized)
-
enumerator FlatData
Primary data storage for GpuIndexFlat (the raw matrix of vectors and vector norms if needed)
-
enumerator IVFLists
Primary data storage for GpuIndexIVF* (the storage for each individual IVF list)
-
enumerator Quantizer
Quantizer (PQ, SQ) dictionary information.
-
enumerator QuantizerPrecomputedCodes
For GpuIndexIVFPQ, “precomputed codes” for more efficient PQ lookup require the use of possibly large tables. These are marked separately from Quantizer as these can frequently be 100s - 1000s of MiB in size
-
enumerator TemporaryMemoryBuffer
StandardGpuResources implementation specific types When using StandardGpuResources, temporary memory allocations (MemorySpace::Temporary) come out of a stack region of memory that is allocated up front for each gpu (e.g., 1.5 GiB upon initialization). This allocation by StandardGpuResources is marked with this AllocType.
-
enumerator TemporaryMemoryOverflow
When using StandardGpuResources, any MemorySpace::Temporary allocations that cannot be satisfied within the TemporaryMemoryBuffer region fall back to calling cudaMalloc which are sized to just the request at hand. These “overflow” temporary allocations are marked with this AllocType.
-
enumerator Other
-
enum MemorySpace
Memory regions accessible to the GPU.
Values:
-
enumerator Temporary
Temporary device memory (guaranteed to no longer be used upon exit of a top-level index call, and where the streams using it have completed GPU work). Typically backed by Device memory (cudaMalloc/cudaFree).
-
enumerator Device
Managed using cudaMalloc/cudaFree (typical GPU device memory)
-
enumerator Unified
Managed using cudaMallocManaged/cudaFree (typical Unified CPU/GPU memory)
-
enumerator Temporary
Functions
-
faiss::Index *index_gpu_to_cpu(const faiss::Index *gpu_index)
converts any GPU index inside gpu_index to a CPU index
-
faiss::Index *index_cpu_to_gpu(GpuResourcesProvider *provider, int device, const faiss::Index *index, const GpuClonerOptions *options = nullptr)
converts any CPU index that can be converted to GPU
-
faiss::Index *index_cpu_to_gpu_multiple(std::vector<GpuResourcesProvider*> &provider, std::vector<int> &devices, const faiss::Index *index, const GpuMultipleClonerOptions *options = nullptr)
-
void bfKnn(GpuResourcesProvider *resources, const GpuDistanceParams &args)
A wrapper for gpu/impl/Distance.cuh to expose direct brute-force k-nearest neighbor searches on an externally-provided region of memory (e.g., from a pytorch tensor). The data (vectors, queries, outDistances, outIndices) can be resident on the GPU or the CPU, but all calculations are performed on the GPU. If the result buffers are on the CPU, results will be copied back when done.
All GPU computation is performed on the current CUDA device, and ordered with respect to resources->getDefaultStreamCurrentDevice().
For each vector in
queries, searches all ofvectorsto find its k nearest neighbors with respect to the given metric
-
void bruteForceKnn(GpuResourcesProvider *resources, faiss::MetricType metric, const float *vectors, bool vectorsRowMajor, int numVectors, const float *queries, bool queriesRowMajor, int numQueries, int dims, int k, float *outDistances, Index::idx_t *outIndices)
Deprecated legacy implementation.
-
std::string allocTypeToString(AllocType t)
Convert an AllocType to string.
-
std::string memorySpaceToString(MemorySpace s)
Convert a MemorySpace to string.
-
AllocInfo makeDevAlloc(AllocType at, cudaStream_t st)
Create an AllocInfo for the current device with MemorySpace::Device.
-
AllocInfo makeTempAlloc(AllocType at, cudaStream_t st)
Create an AllocInfo for the current device with MemorySpace::Temporary.
-
AllocInfo makeSpaceAlloc(AllocType at, MemorySpace sp, cudaStream_t st)
Create an AllocInfo for the current device.
-
void newTestSeed()
Generates and displays a new seed for the test.
-
void setTestSeed(long seed)
Uses an explicit seed for the test.
-
float relativeError(float a, float b)
Returns the relative error in difference between a and b (|a - b| / (0.5 * (|a| + |b|))
-
int randVal(int a, int b)
Generates a random integer in the range [a, b].
-
bool randBool()
Generates a random bool.
-
template<typename T>
T randSelect(std::initializer_list<T> vals) Select a random value from the given list of values provided as an initializer_list
-
std::vector<float> randVecs(size_t num, size_t dim)
Generates a collection of random vectors in the range [0, 1].
-
std::vector<unsigned char> randBinaryVecs(size_t num, size_t dim)
Generates a collection of random bit vectors.
-
void compareIndices(const std::vector<float> &queryVecs, faiss::Index &refIndex, faiss::Index &testIndex, int numQuery, int dim, int k, const std::string &configMsg, float maxRelativeError = 6e-5f, float pctMaxDiff1 = 0.1f, float pctMaxDiffN = 0.005f)
Compare two indices via query for similarity, with a user-specified set of query vectors
-
void compareIndices(faiss::Index &refIndex, faiss::Index &testIndex, int numQuery, int dim, int k, const std::string &configMsg, float maxRelativeError = 6e-5f, float pctMaxDiff1 = 0.1f, float pctMaxDiffN = 0.005f)
Compare two indices via query for similarity, generating random query vectors
-
void compareLists(const float *refDist, const faiss::Index::idx_t *refInd, const float *testDist, const faiss::Index::idx_t *testInd, int dim1, int dim2, const std::string &configMsg, bool printBasicStats, bool printDiffs, bool assertOnErr, float maxRelativeError = 6e-5f, float pctMaxDiff1 = 0.1f, float pctMaxDiffN = 0.005f)
Display specific differences in the two (distance, index) lists.
-
template<typename A, typename B>
void testIVFEquality(A &cpuIndex, B &gpuIndex) Compare IVF lists between a CPU and GPU index.
-
int getCurrentDevice()
Returns the current thread-local GPU device.
-
void setCurrentDevice(int device)
Sets the current thread-local GPU device.
-
int getNumDevices()
Returns the number of available GPU devices.
-
void profilerStart()
Starts the CUDA profiler (exposed via SWIG)
-
void profilerStop()
Stops the CUDA profiler (exposed via SWIG)
-
void synchronizeAllDevices()
Synchronizes the CPU against all devices (equivalent to cudaDeviceSynchronize for each device)
-
const cudaDeviceProp &getDeviceProperties(int device)
Returns a cached cudaDeviceProp for the given device.
-
const cudaDeviceProp &getCurrentDeviceProperties()
Returns the cached cudaDeviceProp for the current device.
-
int getMaxThreads(int device)
Returns the maximum number of threads available for the given GPU device
-
int getMaxThreadsCurrentDevice()
Equivalent to getMaxThreads(getCurrentDevice())
Returns the maximum smem available for the given GPU device.
Equivalent to getMaxSharedMemPerBlock(getCurrentDevice())
-
int getDeviceForAddress(const void *p)
For a given pointer, returns whether or not it is located on a device (deviceId >= 0) or the host (-1).
-
bool getFullUnifiedMemSupport(int device)
Does the given device support full unified memory sharing host memory?
-
bool getFullUnifiedMemSupportCurrentDevice()
Equivalent to getFullUnifiedMemSupport(getCurrentDevice())
-
bool getTensorCoreSupport(int device)
Does the given device support tensor core operations?
-
bool getTensorCoreSupportCurrentDevice()
Equivalent to getTensorCoreSupport(getCurrentDevice())
-
int getMaxKSelection()
Returns the maximum k-selection value supported based on the CUDA SDK that we were compiled with. .cu files can use DeviceDefs.cuh, but this is for non-CUDA files
-
template<typename L1, typename L2>
void streamWaitBase(const L1 &listWaiting, const L2 &listWaitOn) Call for a collection of streams to wait on.
-
template<typename L1>
void streamWait(const L1 &a, const std::initializer_list<cudaStream_t> &b) These versions allow usage of initializer_list as arguments, since otherwise {…} doesn’t have a type
-
template<typename L2>
void streamWait(const std::initializer_list<cudaStream_t> &a, const L2 &b)
-
inline void streamWait(const std::initializer_list<cudaStream_t> &a, const std::initializer_list<cudaStream_t> &b)
-
struct AllocInfo
- #include <GpuResources.h>
Information on what/where an allocation is.
Subclassed by faiss::gpu::AllocRequest
Public Functions
-
inline AllocInfo()
-
inline AllocInfo(AllocType at, int dev, MemorySpace sp, cudaStream_t st)
-
std::string toString() const
Returns a string representation of this info.
Public Members
-
AllocType type
The internal category of the allocation.
-
int device
The device on which the allocation is happening.
-
MemorySpace space
The memory space of the allocation.
-
cudaStream_t stream
The stream on which new work on the memory will be ordered (e.g., if a piece of memory cached and to be returned for this call was last used on stream 3 and a new memory request is for stream 4, the memory manager will synchronize stream 4 to wait for the completion of stream 3 via events or other stream synchronization.
The memory manager guarantees that the returned memory is free to use without data races on this stream specified.
-
inline AllocInfo()
-
struct AllocRequest : public faiss::gpu::AllocInfo
- #include <GpuResources.h>
Information on what/where an allocation is, along with how big it should be.
Public Functions
-
inline AllocRequest()
-
inline AllocRequest(const AllocInfo &info, size_t sz)
-
inline AllocRequest(AllocType at, int dev, MemorySpace sp, cudaStream_t st, size_t sz)
-
std::string toString() const
Returns a string representation of this request.
Public Members
-
size_t size
The size in bytes of the allocation.
-
AllocType type
The internal category of the allocation.
-
int device
The device on which the allocation is happening.
-
MemorySpace space
The memory space of the allocation.
-
cudaStream_t stream
The stream on which new work on the memory will be ordered (e.g., if a piece of memory cached and to be returned for this call was last used on stream 3 and a new memory request is for stream 4, the memory manager will synchronize stream 4 to wait for the completion of stream 3 via events or other stream synchronization.
The memory manager guarantees that the returned memory is free to use without data races on this stream specified.
-
inline AllocRequest()
-
class CpuTimer
- #include <Timer.h>
CPU wallclock elapsed timer.
Public Functions
-
CpuTimer()
Creates and starts a new timer.
-
float elapsedMilliseconds()
Returns elapsed time in milliseconds.
Private Members
-
std::chrono::time_point<std::chrono::steady_clock> start_
-
CpuTimer()
-
class CublasHandleScope
- #include <DeviceUtils.h>
RAII object to manage a cublasHandle_t.
Public Functions
-
CublasHandleScope()
-
~CublasHandleScope()
-
inline cublasHandle_t get()
Private Members
-
cublasHandle_t blasHandle_
-
CublasHandleScope()
-
class CudaEvent
Public Functions
-
explicit CudaEvent(cudaStream_t stream, bool timer = false)
Creates an event and records it in this stream.
-
CudaEvent(const CudaEvent &event) = delete
-
CudaEvent(CudaEvent &&event) noexcept
-
~CudaEvent()
-
inline cudaEvent_t get()
-
void streamWaitOnEvent(cudaStream_t stream)
Wait on this event in this stream.
-
void cpuWaitOnEvent()
Have the CPU wait for the completion of this event.
Private Members
-
cudaEvent_t event_
-
explicit CudaEvent(cudaStream_t stream, bool timer = false)
-
class DeviceScope
- #include <DeviceUtils.h>
RAII object to set the current device, and restore the previous device upon destruction
Public Functions
-
explicit DeviceScope(int device)
-
~DeviceScope()
Private Members
-
int prevDevice_
-
explicit DeviceScope(int device)
-
struct GpuClonerOptions
- #include <GpuClonerOptions.h>
set some options on how to copy to GPU
Subclassed by faiss::gpu::GpuMultipleClonerOptions, faiss::gpu::ToGpuCloner
Public Functions
-
GpuClonerOptions()
Public Members
-
IndicesOptions indicesOptions
how should indices be stored on index types that support indices (anything but GpuIndexFlat*)?
-
bool useFloat16CoarseQuantizer
is the coarse quantizer in float16?
-
bool useFloat16
for GpuIndexIVFFlat, is storage in float16? for GpuIndexIVFPQ, are intermediate calculations in float16?
-
bool usePrecomputed
use precomputed tables?
-
long reserveVecs
reserve vectors in the invfiles?
-
bool storeTransposed
For GpuIndexFlat, store data in transposed layout?
-
bool verbose
Set verbose options on the index.
-
GpuClonerOptions()
-
struct GpuDistanceParams
- #include <GpuDistance.h>
Arguments to brute-force GPU k-nearest neighbor searching.
Public Functions
-
inline GpuDistanceParams()
Public Members
-
faiss::MetricType metric
Search parameter: distance metric.
-
float metricArg
Search parameter: distance metric argument (if applicable) For metric == METRIC_Lp, this is the p-value
-
int k
Search parameter: return k nearest neighbors If the value provided is -1, then we report all pairwise distances without top-k filtering
-
int dims
Vector dimensionality.
-
const void *vectors
If vectorsRowMajor is true, this is numVectors x dims, with dims innermost; otherwise, dims x numVectors, with numVectors innermost
-
DistanceDataType vectorType
-
bool vectorsRowMajor
-
int numVectors
-
const float *vectorNorms
Precomputed L2 norms for each vector in
vectors, which can be optionally provided in advance to speed computation for METRIC_L2
-
const void *queries
If queriesRowMajor is true, this is numQueries x dims, with dims innermost; otherwise, dims x numQueries, with numQueries innermost
-
DistanceDataType queryType
-
bool queriesRowMajor
-
int numQueries
-
float *outDistances
A region of memory size numQueries x k, with k innermost (row major) if k > 0, or if k == -1, a region of memory of size numQueries x numVectors
-
bool ignoreOutDistances
Do we only care about the indices reported, rather than the output distances? Not used if k == -1 (all pairwise distances)
-
IndicesDataType outIndicesType
A region of memory size numQueries x k, with k innermost (row major). Not used if k == -1 (all pairwise distances)
-
void *outIndices
-
inline GpuDistanceParams()
-
class GpuIcmEncoder : public IcmEncoder
- #include <GpuIcmEncoder.h>
Perform LSQ encoding on GPU.
Split input vectors to different devices and call IcmEncoderImpl::encode to encode them
Public Functions
-
GpuIcmEncoder(const LocalSearchQuantizer *lsq, const std::vector<GpuResourcesProvider*> &provs, const std::vector<int> &devices)
-
~GpuIcmEncoder()
-
GpuIcmEncoder(const GpuIcmEncoder&) = delete
-
GpuIcmEncoder &operator=(const GpuIcmEncoder&) = delete
-
void set_binary_term() override
-
void encode(int32_t *codes, const float *x, std::mt19937 &gen, size_t n, size_t ils_iters) const override
Private Members
-
std::unique_ptr<IcmEncoderShards> shards
-
GpuIcmEncoder(const LocalSearchQuantizer *lsq, const std::vector<GpuResourcesProvider*> &provs, const std::vector<int> &devices)
-
struct GpuIcmEncoderFactory : public IcmEncoderFactory
Public Functions
-
explicit GpuIcmEncoderFactory(int ngpus = 1)
-
lsq::IcmEncoder *get(const LocalSearchQuantizer *lsq) override
-
explicit GpuIcmEncoderFactory(int ngpus = 1)
-
class GpuIndex : public faiss::Index
Subclassed by faiss::gpu::GpuIndexFlat, faiss::gpu::GpuIndexIVF
Public Types
-
using idx_t = int64_t
all indices are this type
-
using component_t = float
-
using distance_t = float
Public Functions
-
int getDevice() const
Returns the device that this index is resident on.
-
std::shared_ptr<GpuResources> getResources()
Returns a reference to our GpuResources object that manages memory, stream and handle resources on the GPU
-
void setMinPagingSize(size_t size)
Set the minimum data size for searches (in MiB) for which we use CPU -> GPU paging
-
size_t getMinPagingSize() const
Returns the current minimum data size for paged searches.
-
virtual void add(Index::idx_t, const float *x) override
xcan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void add_with_ids(Index::idx_t n, const float *x, const Index::idx_t *ids) override
xandidscan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void assign(Index::idx_t n, const float *x, Index::idx_t *labels, Index::idx_t k = 1) const override
xandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search(Index::idx_t n, const float *x, Index::idx_t k, float *distances, Index::idx_t *labels) const override
x,distancesandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void compute_residual(const float *x, float *residual, Index::idx_t key) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void compute_residual_n(Index::idx_t n, const float *xs, float *residuals, const Index::idx_t *keys) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reset() = 0
removes all elements from the database.
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
Protected Functions
-
virtual bool addImplRequiresIDs_() const = 0
Does addImpl_ require IDs? If so, and no IDs are provided, we will generate them sequentially based on the order in which the IDs are added
Protected Attributes
-
std::shared_ptr<GpuResources> resources_
Manages streams, cuBLAS handles and scratch memory for devices.
-
const GpuIndexConfig config_
Our configuration options.
-
size_t minPagedSize_
Size above which we page copies from the CPU to GPU.
Private Functions
-
void addPaged_(int n, const float *x, const Index::idx_t *ids)
Handles paged adds if the add set is too large, passes to addImpl_ to actually perform the add for the current page
-
void addPage_(int n, const float *x, const Index::idx_t *ids)
Calls addImpl_ for a single page of GPU-resident data.
-
using idx_t = int64_t
-
class GpuIndexBinaryFlat : public faiss::IndexBinary
- #include <GpuIndexBinaryFlat.h>
A GPU version of IndexBinaryFlat for brute-force comparison of bit vectors via Hamming distance
Public Types
-
using component_t = uint8_t
-
using distance_t = int32_t
Public Functions
-
GpuIndexBinaryFlat(GpuResourcesProvider *resources, const faiss::IndexBinaryFlat *index, GpuIndexBinaryFlatConfig config = GpuIndexBinaryFlatConfig())
Construct from a pre-existing faiss::IndexBinaryFlat instance, copying data over to the given GPU
-
GpuIndexBinaryFlat(GpuResourcesProvider *resources, int dims, GpuIndexBinaryFlatConfig config = GpuIndexBinaryFlatConfig())
Construct an empty instance that can be added to.
-
~GpuIndexBinaryFlat() override
-
int getDevice() const
Returns the device that this index is resident on.
-
std::shared_ptr<GpuResources> getResources()
Returns a reference to our GpuResources object that manages memory, stream and handle resources on the GPU
-
void copyFrom(const faiss::IndexBinaryFlat *index)
Initialize ourselves from the given CPU index; will overwrite all data in ourselves
-
void copyTo(faiss::IndexBinaryFlat *index) const
Copy ourselves to the given CPU index; will overwrite all data in the index instance
-
virtual void add(faiss::IndexBinary::idx_t n, const uint8_t *x) override
Add n vectors of dimension d to the index.
Vectors are implicitly assigned labels ntotal .. ntotal + n - 1
- Parameters:
x – input matrix, size n * d / 8
-
virtual void reset() override
Removes all elements from the database.
-
virtual void search(faiss::IndexBinary::idx_t n, const uint8_t *x, faiss::IndexBinary::idx_t k, int32_t *distances, faiss::IndexBinary::idx_t *labels) const override
Query n vectors of dimension d to the index.
return at most k vectors. If there are not enough results for a query, the result array is padded with -1s.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
distances – output pairwise distances, size n*k
-
virtual void reconstruct(faiss::IndexBinary::idx_t key, uint8_t *recons) const override
Reconstruct a stored vector.
This function may not be defined for some indexes.
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d / 8)
-
virtual void train(idx_t n, const uint8_t *x)
Perform training on a representative set of vectors.
- Parameters:
n – nb of training vectors
x – training vecors, size n * d / 8
-
virtual void add_with_ids(idx_t n, const uint8_t *x, const idx_t *xids)
Same as add, but stores xids instead of sequential ids.
The default implementation fails with an assertion, as it is not supported by all indexes.
- Parameters:
xids – if non-null, ids to store for the vectors (size n)
-
virtual void range_search(idx_t n, const uint8_t *x, int radius, RangeSearchResult *result) const
Query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory). The distances are converted to float to reuse the RangeSearchResult structure, but they are integer. By convention, only distances < radius (strict comparison) are returned, ie. radius = 0 does not return any result and 1 returns only exact same vectors.
- Parameters:
x – input vectors to search, size n * d / 8
radius – search radius
result – result table
-
void assign(idx_t n, const uint8_t *x, idx_t *labels, idx_t k = 1) const
Return the indexes of the k vectors closest to the query x.
This function is identical to search but only returns labels of neighbors.
- Parameters:
x – input vectors to search, size n * d / 8
labels – output labels of the NNs, size n*k
-
virtual size_t remove_ids(const IDSelector &sel)
Removes IDs from the index. Not supported by all indexes.
-
virtual void reconstruct_n(idx_t i0, idx_t ni, uint8_t *recons) const
Reconstruct vectors i0 to i0 + ni - 1.
This function may not be defined for some indexes.
- Parameters:
recons – reconstucted vectors (size ni * d / 8)
-
virtual void search_and_reconstruct(idx_t n, const uint8_t *x, idx_t k, int32_t *distances, idx_t *labels, uint8_t *recons) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting array is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
void display() const
Display the actual class name and some more info.
Public Members
-
int d
vector dimension
-
int code_size
number of bytes per vector ( = d / 8 )
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
Protected Functions
-
void searchFromCpuPaged_(int n, const uint8_t *x, int k, int32_t *outDistancesData, int *outIndicesData) const
Called from search when the input data is on the CPU; potentially allows for pinned memory usage
-
void searchNonPaged_(int n, const uint8_t *x, int k, int32_t *outDistancesData, int *outIndicesData) const
Protected Attributes
-
std::shared_ptr<GpuResources> resources_
Manages streans, cuBLAS handles and scratch memory for devices.
-
const GpuIndexBinaryFlatConfig binaryFlatConfig_
Configuration options.
-
std::unique_ptr<BinaryFlatIndex> data_
Holds our GPU data containing the list of vectors.
-
using component_t = uint8_t
-
struct GpuIndexBinaryFlatConfig : public faiss::gpu::GpuIndexConfig
Public Members
-
int device¶
GPU device on which the index is resident.
-
MemorySpace memorySpace¶
What memory space to use for primary storage. On Pascal and above (CC 6+) architectures, allows GPUs to use more memory than is available on the GPU.
-
int device¶
-
struct GpuIndexConfig
Subclassed by faiss::gpu::GpuIndexBinaryFlatConfig, faiss::gpu::GpuIndexFlatConfig, faiss::gpu::GpuIndexIVFConfig
Public Functions
-
inline GpuIndexConfig()
Public Members
-
int device
GPU device on which the index is resident.
-
MemorySpace memorySpace
What memory space to use for primary storage. On Pascal and above (CC 6+) architectures, allows GPUs to use more memory than is available on the GPU.
-
inline GpuIndexConfig()
-
class GpuIndexFlat : public faiss::gpu::GpuIndex
- #include <GpuIndexFlat.h>
Wrapper around the GPU implementation that looks like faiss::IndexFlat; copies over centroid data from a given faiss::IndexFlat
Subclassed by faiss::gpu::GpuIndexFlatIP, faiss::gpu::GpuIndexFlatL2
Public Types
-
using idx_t = int64_t
all indices are this type
-
using component_t = float
-
using distance_t = float
Public Functions
-
GpuIndexFlat(GpuResourcesProvider *provider, const faiss::IndexFlat *index, GpuIndexFlatConfig config = GpuIndexFlatConfig())
Construct from a pre-existing faiss::IndexFlat instance, copying data over to the given GPU
-
GpuIndexFlat(GpuResourcesProvider *provider, int dims, faiss::MetricType metric, GpuIndexFlatConfig config = GpuIndexFlatConfig())
Construct an empty instance that can be added to.
-
~GpuIndexFlat() override
-
void copyFrom(const faiss::IndexFlat *index)
Initialize ourselves from the given CPU index; will overwrite all data in ourselves
-
void copyTo(faiss::IndexFlat *index) const
Copy ourselves to the given CPU index; will overwrite all data in the index instance
-
size_t getNumVecs() const
Returns the number of vectors we contain.
-
virtual void reset() override
Clears all vectors from this index.
-
virtual void train(Index::idx_t n, const float *x) override
This index is not trained, so this does nothing.
-
virtual void reconstruct(Index::idx_t key, float *out) const override
Reconstruction methods; prefer the batch reconstruct as it will be more efficient
-
virtual void reconstruct_n(Index::idx_t i0, Index::idx_t num, float *out) const override
Batch reconstruction method.
-
virtual void compute_residual(const float *x, float *residual, Index::idx_t key) const override
Compute residual.
-
virtual void compute_residual_n(Index::idx_t n, const float *xs, float *residuals, const Index::idx_t *keys) const override
Compute residual (batch mode)
-
inline FlatIndex *getGpuData()
For internal access.
-
int getDevice() const
Returns the device that this index is resident on.
-
std::shared_ptr<GpuResources> getResources()
Returns a reference to our GpuResources object that manages memory, stream and handle resources on the GPU
-
void setMinPagingSize(size_t size)
Set the minimum data size for searches (in MiB) for which we use CPU -> GPU paging
-
size_t getMinPagingSize() const
Returns the current minimum data size for paged searches.
-
virtual void add_with_ids(Index::idx_t n, const float *x, const Index::idx_t *ids) override
xandidscan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void assign(Index::idx_t n, const float *x, Index::idx_t *labels, Index::idx_t k = 1) const override
xandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search(Index::idx_t n, const float *x, Index::idx_t k, float *distances, Index::idx_t *labels) const override
x,distancesandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
Protected Functions
-
virtual bool addImplRequiresIDs_() const override
Flat index does not require IDs as there is no storage available for them
-
virtual void addImpl_(int n, const float *x, const Index::idx_t *ids) override
Called from GpuIndex for add.
Protected Attributes
-
const GpuIndexFlatConfig flatConfig_
Our configuration options.
-
std::unique_ptr<FlatIndex> data_
Holds our GPU data containing the list of vectors.
-
std::shared_ptr<GpuResources> resources_
Manages streams, cuBLAS handles and scratch memory for devices.
-
const GpuIndexConfig config_
Our configuration options.
-
size_t minPagedSize_
Size above which we page copies from the CPU to GPU.
-
using idx_t = int64_t
-
struct GpuIndexFlatConfig : public faiss::gpu::GpuIndexConfig
Public Functions
-
inline GpuIndexFlatConfig()
Public Members
-
bool useFloat16
Whether or not data is stored as float16.
-
bool storeTransposed
Whether or not data is stored (transparently) in a transposed layout, enabling use of the NN GEMM call, which is ~10% faster. This will improve the speed of the flat index, but will substantially slow down any add() calls made, as all data must be transposed, and will increase storage requirements (we store data in both transposed and non-transposed layouts).
-
int device¶
GPU device on which the index is resident.
-
MemorySpace memorySpace¶
What memory space to use for primary storage. On Pascal and above (CC 6+) architectures, allows GPUs to use more memory than is available on the GPU.
-
inline GpuIndexFlatConfig()
-
class GpuIndexFlatIP : public faiss::gpu::GpuIndexFlat
- #include <GpuIndexFlat.h>
Wrapper around the GPU implementation that looks like faiss::IndexFlatIP; copies over centroid data from a given faiss::IndexFlat
Public Types
-
using idx_t = int64_t
all indices are this type
-
using component_t = float
-
using distance_t = float
Public Functions
-
GpuIndexFlatIP(GpuResourcesProvider *provider, faiss::IndexFlatIP *index, GpuIndexFlatConfig config = GpuIndexFlatConfig())
Construct from a pre-existing faiss::IndexFlatIP instance, copying data over to the given GPU
-
GpuIndexFlatIP(GpuResourcesProvider *provider, int dims, GpuIndexFlatConfig config = GpuIndexFlatConfig())
Construct an empty instance that can be added to.
-
void copyFrom(faiss::IndexFlat *index)
Initialize ourselves from the given CPU index; will overwrite all data in ourselves
-
void copyTo(faiss::IndexFlat *index)
Copy ourselves to the given CPU index; will overwrite all data in the index instance
-
void copyFrom(const faiss::IndexFlat *index)
Initialize ourselves from the given CPU index; will overwrite all data in ourselves
-
void copyTo(faiss::IndexFlat *index) const
Copy ourselves to the given CPU index; will overwrite all data in the index instance
-
size_t getNumVecs() const
Returns the number of vectors we contain.
-
virtual void reset() override
Clears all vectors from this index.
-
virtual void train(Index::idx_t n, const float *x) override
This index is not trained, so this does nothing.
-
virtual void reconstruct(Index::idx_t key, float *out) const override
Reconstruction methods; prefer the batch reconstruct as it will be more efficient
-
virtual void reconstruct_n(Index::idx_t i0, Index::idx_t num, float *out) const override
Batch reconstruction method.
-
virtual void compute_residual(const float *x, float *residual, Index::idx_t key) const override
Compute residual.
-
virtual void compute_residual_n(Index::idx_t n, const float *xs, float *residuals, const Index::idx_t *keys) const override
Compute residual (batch mode)
-
inline FlatIndex *getGpuData()
For internal access.
-
int getDevice() const
Returns the device that this index is resident on.
-
std::shared_ptr<GpuResources> getResources()
Returns a reference to our GpuResources object that manages memory, stream and handle resources on the GPU
-
void setMinPagingSize(size_t size)
Set the minimum data size for searches (in MiB) for which we use CPU -> GPU paging
-
size_t getMinPagingSize() const
Returns the current minimum data size for paged searches.
-
virtual void add_with_ids(Index::idx_t n, const float *x, const Index::idx_t *ids) override
xandidscan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void assign(Index::idx_t n, const float *x, Index::idx_t *labels, Index::idx_t k = 1) const override
xandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search(Index::idx_t n, const float *x, Index::idx_t k, float *distances, Index::idx_t *labels) const override
x,distancesandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
Protected Functions
-
virtual bool addImplRequiresIDs_() const override
Flat index does not require IDs as there is no storage available for them
Protected Attributes
-
const GpuIndexFlatConfig flatConfig_
Our configuration options.
-
std::unique_ptr<FlatIndex> data_
Holds our GPU data containing the list of vectors.
-
std::shared_ptr<GpuResources> resources_
Manages streams, cuBLAS handles and scratch memory for devices.
-
const GpuIndexConfig config_
Our configuration options.
-
size_t minPagedSize_
Size above which we page copies from the CPU to GPU.
-
using idx_t = int64_t
-
class GpuIndexFlatL2 : public faiss::gpu::GpuIndexFlat
- #include <GpuIndexFlat.h>
Wrapper around the GPU implementation that looks like faiss::IndexFlatL2; copies over centroid data from a given faiss::IndexFlat
Public Types
-
using idx_t = int64_t
all indices are this type
-
using component_t = float
-
using distance_t = float
Public Functions
-
GpuIndexFlatL2(GpuResourcesProvider *provider, faiss::IndexFlatL2 *index, GpuIndexFlatConfig config = GpuIndexFlatConfig())
Construct from a pre-existing faiss::IndexFlatL2 instance, copying data over to the given GPU
-
GpuIndexFlatL2(GpuResourcesProvider *provider, int dims, GpuIndexFlatConfig config = GpuIndexFlatConfig())
Construct an empty instance that can be added to.
-
void copyFrom(faiss::IndexFlat *index)
Initialize ourselves from the given CPU index; will overwrite all data in ourselves
-
void copyTo(faiss::IndexFlat *index)
Copy ourselves to the given CPU index; will overwrite all data in the index instance
-
void copyFrom(const faiss::IndexFlat *index)
Initialize ourselves from the given CPU index; will overwrite all data in ourselves
-
void copyTo(faiss::IndexFlat *index) const
Copy ourselves to the given CPU index; will overwrite all data in the index instance
-
size_t getNumVecs() const
Returns the number of vectors we contain.
-
virtual void reset() override
Clears all vectors from this index.
-
virtual void train(Index::idx_t n, const float *x) override
This index is not trained, so this does nothing.
-
virtual void reconstruct(Index::idx_t key, float *out) const override
Reconstruction methods; prefer the batch reconstruct as it will be more efficient
-
virtual void reconstruct_n(Index::idx_t i0, Index::idx_t num, float *out) const override
Batch reconstruction method.
-
virtual void compute_residual(const float *x, float *residual, Index::idx_t key) const override
Compute residual.
-
virtual void compute_residual_n(Index::idx_t n, const float *xs, float *residuals, const Index::idx_t *keys) const override
Compute residual (batch mode)
-
inline FlatIndex *getGpuData()
For internal access.
-
int getDevice() const
Returns the device that this index is resident on.
-
std::shared_ptr<GpuResources> getResources()
Returns a reference to our GpuResources object that manages memory, stream and handle resources on the GPU
-
void setMinPagingSize(size_t size)
Set the minimum data size for searches (in MiB) for which we use CPU -> GPU paging
-
size_t getMinPagingSize() const
Returns the current minimum data size for paged searches.
-
virtual void add_with_ids(Index::idx_t n, const float *x, const Index::idx_t *ids) override
xandidscan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void assign(Index::idx_t n, const float *x, Index::idx_t *labels, Index::idx_t k = 1) const override
xandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search(Index::idx_t n, const float *x, Index::idx_t k, float *distances, Index::idx_t *labels) const override
x,distancesandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
Protected Functions
-
virtual bool addImplRequiresIDs_() const override
Flat index does not require IDs as there is no storage available for them
Protected Attributes
-
const GpuIndexFlatConfig flatConfig_
Our configuration options.
-
std::unique_ptr<FlatIndex> data_
Holds our GPU data containing the list of vectors.
-
std::shared_ptr<GpuResources> resources_
Manages streams, cuBLAS handles and scratch memory for devices.
-
const GpuIndexConfig config_
Our configuration options.
-
size_t minPagedSize_
Size above which we page copies from the CPU to GPU.
-
using idx_t = int64_t
-
class GpuIndexIVF : public faiss::gpu::GpuIndex
Subclassed by faiss::gpu::GpuIndexIVFFlat, faiss::gpu::GpuIndexIVFPQ, faiss::gpu::GpuIndexIVFScalarQuantizer
Public Types
-
using idx_t = int64_t
all indices are this type
-
using component_t = float
-
using distance_t = float
Public Functions
-
GpuIndexIVF(GpuResourcesProvider *provider, int dims, faiss::MetricType metric, float metricArg, int nlist, GpuIndexIVFConfig config = GpuIndexIVFConfig())
-
~GpuIndexIVF() override
-
int getNumLists() const
Returns the number of inverted lists we’re managing.
-
virtual int getListLength(int listId) const = 0
Returns the number of vectors present in a particular inverted list.
-
virtual std::vector<uint8_t> getListVectorData(int listId, bool gpuFormat = false) const = 0
Return the encoded vector data contained in a particular inverted list, for debugging purposes. If gpuFormat is true, the data is returned as it is encoded in the GPU-side representation. Otherwise, it is converted to the CPU format. compliant format, while the native GPU format may differ.
-
virtual std::vector<Index::idx_t> getListIndices(int listId) const = 0
Return the vector indices contained in a particular inverted list, for debugging purposes.
-
GpuIndexFlat *getQuantizer()
Return the quantizer we’re using.
-
void setNumProbes(int nprobe)
Sets the number of list probes per query.
-
int getNumProbes() const
Returns our current number of list probes per query.
-
int getDevice() const
Returns the device that this index is resident on.
-
std::shared_ptr<GpuResources> getResources()
Returns a reference to our GpuResources object that manages memory, stream and handle resources on the GPU
-
void setMinPagingSize(size_t size)
Set the minimum data size for searches (in MiB) for which we use CPU -> GPU paging
-
size_t getMinPagingSize() const
Returns the current minimum data size for paged searches.
-
virtual void add(Index::idx_t, const float *x) override
xcan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void add_with_ids(Index::idx_t n, const float *x, const Index::idx_t *ids) override
xandidscan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void assign(Index::idx_t n, const float *x, Index::idx_t *labels, Index::idx_t k = 1) const override
xandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search(Index::idx_t n, const float *x, Index::idx_t k, float *distances, Index::idx_t *labels) const override
x,distancesandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void compute_residual(const float *x, float *residual, Index::idx_t key) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void compute_residual_n(Index::idx_t n, const float *xs, float *residuals, const Index::idx_t *keys) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void train(idx_t n, const float *x)
Perform training on a representative set of vectors
- Parameters:
n – nb of training vectors
x – training vecors, size n * d
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual void reset() = 0
removes all elements from the database.
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
ClusteringParameters cp
Exposing this like the CPU version for manipulation.
-
int nlist
Exposing this like the CPU version for query.
-
int nprobe
Exposing this like the CPU version for manipulation.
-
GpuIndexFlat *quantizer
Exposeing this like the CPU version for query.
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
Protected Functions
-
virtual bool addImplRequiresIDs_() const override
Does addImpl_ require IDs? If so, and no IDs are provided, we will generate them sequentially based on the order in which the IDs are added
Protected Attributes
-
const GpuIndexIVFConfig ivfConfig_
Our configuration options.
-
std::shared_ptr<GpuResources> resources_
Manages streams, cuBLAS handles and scratch memory for devices.
-
const GpuIndexConfig config_
Our configuration options.
-
size_t minPagedSize_
Size above which we page copies from the CPU to GPU.
Private Functions
-
void init_()
Shared initialization functions.
-
using idx_t = int64_t
-
struct GpuIndexIVFConfig : public faiss::gpu::GpuIndexConfig
Subclassed by faiss::gpu::GpuIndexIVFFlatConfig, faiss::gpu::GpuIndexIVFPQConfig, faiss::gpu::GpuIndexIVFScalarQuantizerConfig
Public Functions
-
inline GpuIndexIVFConfig()
Public Members
-
IndicesOptions indicesOptions
Index storage options for the GPU.
-
GpuIndexFlatConfig flatConfig
Configuration for the coarse quantizer object.
-
int device¶
GPU device on which the index is resident.
-
MemorySpace memorySpace¶
What memory space to use for primary storage. On Pascal and above (CC 6+) architectures, allows GPUs to use more memory than is available on the GPU.
-
inline GpuIndexIVFConfig()
-
class GpuIndexIVFFlat : public faiss::gpu::GpuIndexIVF
- #include <GpuIndexIVFFlat.h>
Wrapper around the GPU implementation that looks like faiss::IndexIVFFlat
Public Types
-
using idx_t = int64_t
all indices are this type
-
using component_t = float
-
using distance_t = float
Public Functions
-
GpuIndexIVFFlat(GpuResourcesProvider *provider, const faiss::IndexIVFFlat *index, GpuIndexIVFFlatConfig config = GpuIndexIVFFlatConfig())
Construct from a pre-existing faiss::IndexIVFFlat instance, copying data over to the given GPU, if the input index is trained.
-
GpuIndexIVFFlat(GpuResourcesProvider *provider, int dims, int nlist, faiss::MetricType metric, GpuIndexIVFFlatConfig config = GpuIndexIVFFlatConfig())
Constructs a new instance with an empty flat quantizer; the user provides the number of lists desired.
-
~GpuIndexIVFFlat() override
-
void reserveMemory(size_t numVecs)
Reserve GPU memory in our inverted lists for this number of vectors.
-
void copyFrom(const faiss::IndexIVFFlat *index)
Initialize ourselves from the given CPU index; will overwrite all data in ourselves
-
void copyTo(faiss::IndexIVFFlat *index) const
Copy ourselves to the given CPU index; will overwrite all data in the index instance
-
size_t reclaimMemory()
After adding vectors, one can call this to reclaim device memory to exactly the amount needed. Returns space reclaimed in bytes
-
virtual void reset() override
Clears out all inverted lists, but retains the coarse centroid information
-
virtual void train(Index::idx_t n, const float *x) override
Trains the coarse quantizer based on the given vector data.
-
virtual int getListLength(int listId) const override
Returns the number of vectors present in a particular inverted list.
-
virtual std::vector<uint8_t> getListVectorData(int listId, bool gpuFormat = false) const override
Return the encoded vector data contained in a particular inverted list, for debugging purposes. If gpuFormat is true, the data is returned as it is encoded in the GPU-side representation. Otherwise, it is converted to the CPU format. compliant format, while the native GPU format may differ.
-
virtual std::vector<Index::idx_t> getListIndices(int listId) const override
Return the vector indices contained in a particular inverted list, for debugging purposes.
-
int getNumLists() const
Returns the number of inverted lists we’re managing.
-
GpuIndexFlat *getQuantizer()
Return the quantizer we’re using.
-
void setNumProbes(int nprobe)
Sets the number of list probes per query.
-
int getNumProbes() const
Returns our current number of list probes per query.
-
int getDevice() const
Returns the device that this index is resident on.
-
std::shared_ptr<GpuResources> getResources()
Returns a reference to our GpuResources object that manages memory, stream and handle resources on the GPU
-
void setMinPagingSize(size_t size)
Set the minimum data size for searches (in MiB) for which we use CPU -> GPU paging
-
size_t getMinPagingSize() const
Returns the current minimum data size for paged searches.
-
virtual void add(Index::idx_t, const float *x) override
xcan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void add_with_ids(Index::idx_t n, const float *x, const Index::idx_t *ids) override
xandidscan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void assign(Index::idx_t n, const float *x, Index::idx_t *labels, Index::idx_t k = 1) const override
xandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search(Index::idx_t n, const float *x, Index::idx_t k, float *distances, Index::idx_t *labels) const override
x,distancesandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void compute_residual(const float *x, float *residual, Index::idx_t key) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void compute_residual_n(Index::idx_t n, const float *xs, float *residuals, const Index::idx_t *keys) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
ClusteringParameters cp
Exposing this like the CPU version for manipulation.
-
int nlist
Exposing this like the CPU version for query.
-
int nprobe
Exposing this like the CPU version for manipulation.
-
GpuIndexFlat *quantizer
Exposeing this like the CPU version for query.
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
Protected Functions
-
virtual void addImpl_(int n, const float *x, const Index::idx_t *ids) override
Called from GpuIndex for add/add_with_ids.
-
virtual void searchImpl_(int n, const float *x, int k, float *distances, Index::idx_t *labels) const override
Called from GpuIndex for search.
-
virtual bool addImplRequiresIDs_() const override
Does addImpl_ require IDs? If so, and no IDs are provided, we will generate them sequentially based on the order in which the IDs are added
Protected Attributes
-
const GpuIndexIVFFlatConfig ivfFlatConfig_
Our configuration options.
-
size_t reserveMemoryVecs_
Desired inverted list memory reservation.
-
std::unique_ptr<IVFFlat> index_
Instance that we own; contains the inverted list.
-
const GpuIndexIVFConfig ivfConfig_
Our configuration options.
-
std::shared_ptr<GpuResources> resources_
Manages streams, cuBLAS handles and scratch memory for devices.
-
const GpuIndexConfig config_
Our configuration options.
-
size_t minPagedSize_
Size above which we page copies from the CPU to GPU.
-
using idx_t = int64_t
-
struct GpuIndexIVFFlatConfig : public faiss::gpu::GpuIndexIVFConfig
Public Functions
-
inline GpuIndexIVFFlatConfig()
Public Members
-
bool interleavedLayout
Use the alternative memory layout for the IVF lists (currently the default)
-
IndicesOptions indicesOptions¶
Index storage options for the GPU.
-
GpuIndexFlatConfig flatConfig¶
Configuration for the coarse quantizer object.
-
int device¶
GPU device on which the index is resident.
-
MemorySpace memorySpace¶
What memory space to use for primary storage. On Pascal and above (CC 6+) architectures, allows GPUs to use more memory than is available on the GPU.
-
inline GpuIndexIVFFlatConfig()
-
class GpuIndexIVFPQ : public faiss::gpu::GpuIndexIVF
- #include <GpuIndexIVFPQ.h>
IVFPQ index for the GPU.
Public Types
-
using idx_t = int64_t
all indices are this type
-
using component_t = float
-
using distance_t = float
Public Functions
-
GpuIndexIVFPQ(GpuResourcesProvider *provider, const faiss::IndexIVFPQ *index, GpuIndexIVFPQConfig config = GpuIndexIVFPQConfig())
Construct from a pre-existing faiss::IndexIVFPQ instance, copying data over to the given GPU, if the input index is trained.
-
GpuIndexIVFPQ(GpuResourcesProvider *provider, int dims, int nlist, int subQuantizers, int bitsPerCode, faiss::MetricType metric, GpuIndexIVFPQConfig config = GpuIndexIVFPQConfig())
Construct an empty index.
-
~GpuIndexIVFPQ() override
-
void copyFrom(const faiss::IndexIVFPQ *index)
Reserve space on the GPU for the inverted lists for
numvectors, assumed equally distributed among Initialize ourselves from the given CPU index; will overwrite all data in ourselves
-
void copyTo(faiss::IndexIVFPQ *index) const
Copy ourselves to the given CPU index; will overwrite all data in the index instance
-
void reserveMemory(size_t numVecs)
Reserve GPU memory in our inverted lists for this number of vectors.
-
void setPrecomputedCodes(bool enable)
Enable or disable pre-computed codes.
-
bool getPrecomputedCodes() const
Are pre-computed codes enabled?
-
int getNumSubQuantizers() const
Return the number of sub-quantizers we are using.
-
int getBitsPerCode() const
Return the number of bits per PQ code.
-
int getCentroidsPerSubQuantizer() const
Return the number of centroids per PQ code (2^bits per code)
-
size_t reclaimMemory()
After adding vectors, one can call this to reclaim device memory to exactly the amount needed. Returns space reclaimed in bytes
-
virtual void reset() override
Clears out all inverted lists, but retains the coarse and product centroid information
-
virtual void train(Index::idx_t n, const float *x) override
Trains the coarse and product quantizer based on the given vector data.
-
virtual int getListLength(int listId) const override
Returns the number of vectors present in a particular inverted list.
-
virtual std::vector<uint8_t> getListVectorData(int listId, bool gpuFormat = false) const override
Return the encoded vector data contained in a particular inverted list, for debugging purposes. If gpuFormat is true, the data is returned as it is encoded in the GPU-side representation. Otherwise, it is converted to the CPU format. compliant format, while the native GPU format may differ.
-
virtual std::vector<Index::idx_t> getListIndices(int listId) const override
Return the vector indices contained in a particular inverted list, for debugging purposes.
-
int getNumLists() const
Returns the number of inverted lists we’re managing.
-
GpuIndexFlat *getQuantizer()
Return the quantizer we’re using.
-
void setNumProbes(int nprobe)
Sets the number of list probes per query.
-
int getNumProbes() const
Returns our current number of list probes per query.
-
int getDevice() const
Returns the device that this index is resident on.
-
std::shared_ptr<GpuResources> getResources()
Returns a reference to our GpuResources object that manages memory, stream and handle resources on the GPU
-
void setMinPagingSize(size_t size)
Set the minimum data size for searches (in MiB) for which we use CPU -> GPU paging
-
size_t getMinPagingSize() const
Returns the current minimum data size for paged searches.
-
virtual void add(Index::idx_t, const float *x) override
xcan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void add_with_ids(Index::idx_t n, const float *x, const Index::idx_t *ids) override
xandidscan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void assign(Index::idx_t n, const float *x, Index::idx_t *labels, Index::idx_t k = 1) const override
xandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search(Index::idx_t n, const float *x, Index::idx_t k, float *distances, Index::idx_t *labels) const override
x,distancesandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void compute_residual(const float *x, float *residual, Index::idx_t key) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void compute_residual_n(Index::idx_t n, const float *xs, float *residuals, const Index::idx_t *keys) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
ProductQuantizer pq
Like the CPU version, we expose a publically-visible ProductQuantizer for manipulation
-
ClusteringParameters cp
Exposing this like the CPU version for manipulation.
-
int nlist
Exposing this like the CPU version for query.
-
int nprobe
Exposing this like the CPU version for manipulation.
-
GpuIndexFlat *quantizer
Exposeing this like the CPU version for query.
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
Protected Functions
-
virtual void addImpl_(int n, const float *x, const Index::idx_t *ids) override
Called from GpuIndex for add/add_with_ids.
-
virtual void searchImpl_(int n, const float *x, int k, float *distances, Index::idx_t *labels) const override
Called from GpuIndex for search.
-
void verifySettings_() const
Throws errors if configuration settings are improper.
-
void trainResidualQuantizer_(Index::idx_t n, const float *x)
Trains the PQ quantizer based on the given vector data.
-
virtual bool addImplRequiresIDs_() const override
Does addImpl_ require IDs? If so, and no IDs are provided, we will generate them sequentially based on the order in which the IDs are added
Protected Attributes
-
const GpuIndexIVFPQConfig ivfpqConfig_
Our configuration options that we were initialized with.
-
bool usePrecomputedTables_
Runtime override: whether or not we use precomputed tables.
-
int subQuantizers_
Number of sub-quantizers per encoded vector.
-
int bitsPerCode_
Bits per sub-quantizer code.
-
size_t reserveMemoryVecs_
Desired inverted list memory reservation.
-
std::unique_ptr<IVFPQ> index_
The product quantizer instance that we own; contains the inverted lists
-
const GpuIndexIVFConfig ivfConfig_
Our configuration options.
-
std::shared_ptr<GpuResources> resources_
Manages streams, cuBLAS handles and scratch memory for devices.
-
const GpuIndexConfig config_
Our configuration options.
-
size_t minPagedSize_
Size above which we page copies from the CPU to GPU.
-
using idx_t = int64_t
-
struct GpuIndexIVFPQConfig : public faiss::gpu::GpuIndexIVFConfig
Public Functions
-
inline GpuIndexIVFPQConfig()
Public Members
-
bool useFloat16LookupTables
Whether or not float16 residual distance tables are used in the list scanning kernels. When subQuantizers * 2^bitsPerCode > 16384, this is required.
-
bool usePrecomputedTables
Whether or not we enable the precomputed table option for search, which can substantially increase the memory requirement.
-
bool interleavedLayout
Use the alternative memory layout for the IVF lists WARNING: this is a feature under development, do not use!
-
bool useMMCodeDistance
Use GEMM-backed computation of PQ code distances for the no precomputed table version of IVFPQ. This is for debugging purposes, it should not substantially affect the results one way for another.
Note that MM code distance is enabled automatically if one uses a number of dimensions per sub-quantizer that is not natively specialized (an odd number like 7 or so).
-
IndicesOptions indicesOptions¶
Index storage options for the GPU.
-
GpuIndexFlatConfig flatConfig¶
Configuration for the coarse quantizer object.
-
int device¶
GPU device on which the index is resident.
-
MemorySpace memorySpace¶
What memory space to use for primary storage. On Pascal and above (CC 6+) architectures, allows GPUs to use more memory than is available on the GPU.
-
inline GpuIndexIVFPQConfig()
-
class GpuIndexIVFScalarQuantizer : public faiss::gpu::GpuIndexIVF
- #include <GpuIndexIVFScalarQuantizer.h>
Wrapper around the GPU implementation that looks like faiss::IndexIVFScalarQuantizer
Public Types
-
using idx_t = int64_t
all indices are this type
-
using component_t = float
-
using distance_t = float
Public Functions
-
GpuIndexIVFScalarQuantizer(GpuResourcesProvider *provider, const faiss::IndexIVFScalarQuantizer *index, GpuIndexIVFScalarQuantizerConfig config = GpuIndexIVFScalarQuantizerConfig())
Construct from a pre-existing faiss::IndexIVFScalarQuantizer instance, copying data over to the given GPU, if the input index is trained.
-
GpuIndexIVFScalarQuantizer(GpuResourcesProvider *provider, int dims, int nlist, faiss::ScalarQuantizer::QuantizerType qtype, faiss::MetricType metric = MetricType::METRIC_L2, bool encodeResidual = true, GpuIndexIVFScalarQuantizerConfig config = GpuIndexIVFScalarQuantizerConfig())
Constructs a new instance with an empty flat quantizer; the user provides the number of lists desired.
-
~GpuIndexIVFScalarQuantizer() override
-
void reserveMemory(size_t numVecs)
Reserve GPU memory in our inverted lists for this number of vectors.
-
void copyFrom(const faiss::IndexIVFScalarQuantizer *index)
Initialize ourselves from the given CPU index; will overwrite all data in ourselves
-
void copyTo(faiss::IndexIVFScalarQuantizer *index) const
Copy ourselves to the given CPU index; will overwrite all data in the index instance
-
size_t reclaimMemory()
After adding vectors, one can call this to reclaim device memory to exactly the amount needed. Returns space reclaimed in bytes
-
virtual void reset() override
Clears out all inverted lists, but retains the coarse and scalar quantizer information
-
virtual void train(Index::idx_t n, const float *x) override
Trains the coarse and scalar quantizer based on the given vector data.
-
virtual int getListLength(int listId) const override
Returns the number of vectors present in a particular inverted list.
-
virtual std::vector<uint8_t> getListVectorData(int listId, bool gpuFormat = false) const override
Return the encoded vector data contained in a particular inverted list, for debugging purposes. If gpuFormat is true, the data is returned as it is encoded in the GPU-side representation. Otherwise, it is converted to the CPU format. compliant format, while the native GPU format may differ.
-
virtual std::vector<Index::idx_t> getListIndices(int listId) const override
Return the vector indices contained in a particular inverted list, for debugging purposes.
-
int getNumLists() const
Returns the number of inverted lists we’re managing.
-
GpuIndexFlat *getQuantizer()
Return the quantizer we’re using.
-
void setNumProbes(int nprobe)
Sets the number of list probes per query.
-
int getNumProbes() const
Returns our current number of list probes per query.
-
int getDevice() const
Returns the device that this index is resident on.
-
std::shared_ptr<GpuResources> getResources()
Returns a reference to our GpuResources object that manages memory, stream and handle resources on the GPU
-
void setMinPagingSize(size_t size)
Set the minimum data size for searches (in MiB) for which we use CPU -> GPU paging
-
size_t getMinPagingSize() const
Returns the current minimum data size for paged searches.
-
virtual void add(Index::idx_t, const float *x) override
xcan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void add_with_ids(Index::idx_t n, const float *x, const Index::idx_t *ids) override
xandidscan be resident on the CPU or any GPU; copies are performed as needed Handles paged adds if the add set is too large; calls addInternal_
-
virtual void assign(Index::idx_t n, const float *x, Index::idx_t *labels, Index::idx_t k = 1) const override
xandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void search(Index::idx_t n, const float *x, Index::idx_t k, float *distances, Index::idx_t *labels) const override
x,distancesandlabelscan be resident on the CPU or any GPU; copies are performed as needed
-
virtual void compute_residual(const float *x, float *residual, Index::idx_t key) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void compute_residual_n(Index::idx_t n, const float *xs, float *residuals, const Index::idx_t *keys) const override
Overridden to force GPU indices to provide their own GPU-friendly implementation
-
virtual void range_search(idx_t n, const float *x, float radius, RangeSearchResult *result) const
query n vectors of dimension d to the index.
return all vectors with distance < radius. Note that many indexes do not implement the range_search (only the k-NN search is mandatory).
- Parameters:
x – input vectors to search, size n * d
radius – search radius
result – result table
-
virtual size_t remove_ids(const IDSelector &sel)
removes IDs from the index. Not supported by all indexes. Returns the number of elements removed.
-
virtual void reconstruct(idx_t key, float *recons) const
Reconstruct a stored vector (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
key – id of the vector to reconstruct
recons – reconstucted vector (size d)
-
virtual void reconstruct_batch(idx_t n, const idx_t *keys, float *recons) const
Reconstruct several stored vectors (or an approximation if lossy coding)
this function may not be defined for some indexes
- Parameters:
n – number of vectors to reconstruct
keys – ids of the vectors to reconstruct (size n)
recons – reconstucted vector (size n * d)
-
virtual void reconstruct_n(idx_t i0, idx_t ni, float *recons) const
Reconstruct vectors i0 to i0 + ni - 1
this function may not be defined for some indexes
- Parameters:
recons – reconstucted vector (size ni * d)
-
virtual void search_and_reconstruct(idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, float *recons) const
Similar to search, but also reconstructs the stored vectors (or an approximation in the case of lossy coding) for the search results.
If there are not enough results for a query, the resulting arrays is padded with -1s.
- Parameters:
recons – reconstructed vectors size (n, k, d)
-
virtual DistanceComputer *get_distance_computer() const
Get a DistanceComputer (defined in AuxIndexStructures) object for this kind of index.
DistanceComputer is implemented for indexes that support random access of their vectors.
-
virtual size_t sa_code_size() const
size of the produced codes in bytes
-
virtual void sa_encode(idx_t n, const float *x, uint8_t *bytes) const
encode a set of vectors
- Parameters:
n – number of vectors
x – input vectors, size n * d
bytes – output encoded vectors, size n * sa_code_size()
-
virtual void sa_decode(idx_t n, const uint8_t *bytes, float *x) const
decode a set of vectors
- Parameters:
n – number of vectors
bytes – input encoded vectors, size n * sa_code_size()
x – output vectors, size n * d
Public Members
-
faiss::ScalarQuantizer sq
Exposed like the CPU version.
-
bool by_residual
Exposed like the CPU version.
-
ClusteringParameters cp
Exposing this like the CPU version for manipulation.
-
int nlist
Exposing this like the CPU version for query.
-
int nprobe
Exposing this like the CPU version for manipulation.
-
GpuIndexFlat *quantizer
Exposeing this like the CPU version for query.
-
int d
vector dimension
-
idx_t ntotal
total nb of indexed vectors
-
bool verbose
verbosity level
-
bool is_trained
set if the Index does not require training, or if training is done already
-
MetricType metric_type
type of metric this index uses for search
-
float metric_arg
argument of the metric type
Protected Functions
-
virtual void addImpl_(int n, const float *x, const Index::idx_t *ids) override
Called from GpuIndex for add/add_with_ids.
-
virtual void searchImpl_(int n, const float *x, int k, float *distances, Index::idx_t *labels) const override
Called from GpuIndex for search.
-
void trainResiduals_(Index::idx_t n, const float *x)
Called from train to handle SQ residual training.
-
virtual bool addImplRequiresIDs_() const override
Does addImpl_ require IDs? If so, and no IDs are provided, we will generate them sequentially based on the order in which the IDs are added
Protected Attributes
-
const GpuIndexIVFScalarQuantizerConfig ivfSQConfig_
Our configuration options.
-
size_t reserveMemoryVecs_
Desired inverted list memory reservation.
-
std::unique_ptr<IVFFlat> index_
Instance that we own; contains the inverted list.
-
const GpuIndexIVFConfig ivfConfig_
Our configuration options.
-
std::shared_ptr<GpuResources> resources_
Manages streams, cuBLAS handles and scratch memory for devices.
-
const GpuIndexConfig config_
Our configuration options.
-
size_t minPagedSize_
Size above which we page copies from the CPU to GPU.
-
using idx_t = int64_t
-
struct GpuIndexIVFScalarQuantizerConfig : public faiss::gpu::GpuIndexIVFConfig
Public Functions
-
inline GpuIndexIVFScalarQuantizerConfig()
Public Members
-
bool interleavedLayout
Use the alternative memory layout for the IVF lists (currently the default)
-
IndicesOptions indicesOptions¶
Index storage options for the GPU.
-
GpuIndexFlatConfig flatConfig¶
Configuration for the coarse quantizer object.
-
int device¶
GPU device on which the index is resident.
-
MemorySpace memorySpace¶
What memory space to use for primary storage. On Pascal and above (CC 6+) architectures, allows GPUs to use more memory than is available on the GPU.
-
inline GpuIndexIVFScalarQuantizerConfig()
-
struct GpuMemoryReservation
- #include <GpuResources.h>
A RAII object that manages a temporary memory request.
Public Functions
-
GpuMemoryReservation()
-
GpuMemoryReservation(GpuResources *r, int dev, cudaStream_t str, void *p, size_t sz)
-
GpuMemoryReservation(GpuMemoryReservation &&m) noexcept
-
~GpuMemoryReservation()
-
GpuMemoryReservation &operator=(GpuMemoryReservation &&m)
-
inline void *get()
-
void release()
-
GpuMemoryReservation()
-
struct GpuMultipleClonerOptions : public faiss::gpu::GpuClonerOptions
Subclassed by faiss::gpu::ToGpuClonerMultiple
Public Functions
-
GpuMultipleClonerOptions()
Public Members
-
bool shard
Whether to shard the index across GPUs, versus replication across GPUs
-
int shard_type
IndexIVF::copy_subset_to subset type.
-
IndicesOptions indicesOptions
how should indices be stored on index types that support indices (anything but GpuIndexFlat*)?
-
bool useFloat16CoarseQuantizer
is the coarse quantizer in float16?
-
bool useFloat16
for GpuIndexIVFFlat, is storage in float16? for GpuIndexIVFPQ, are intermediate calculations in float16?
-
bool usePrecomputed
use precomputed tables?
-
long reserveVecs
reserve vectors in the invfiles?
-
bool storeTransposed
For GpuIndexFlat, store data in transposed layout?
-
bool verbose
Set verbose options on the index.
-
GpuMultipleClonerOptions()
-
struct GpuParameterSpace : public faiss::ParameterSpace
- #include <GpuAutoTune.h>
parameter space and setters for GPU indexes
Public Functions
-
virtual void initialize(const faiss::Index *index) override
initialize with reasonable parameters for the index
-
virtual void set_index_parameter(faiss::Index *index, const std::string &name, double val) const override
set a combination of parameters on an index
-
size_t n_combinations() const¶
nb of combinations, = product of values sizes
-
bool combination_ge(size_t c1, size_t c2) const¶
returns whether combinations c1 >= c2 in the tuple sense
-
std::string combination_name(size_t cno) const¶
get string representation of the combination
-
void display() const¶
print a description on stdout
-
ParameterRange &add_range(const std::string &name)¶
add a new parameter (or return it if it exists)
-
void set_index_parameters(Index *index, size_t cno) const¶
set a combination of parameters on an index
-
void set_index_parameters(Index *index, const char *param_string) const¶
set a combination of parameters described by a string
-
void update_bounds(size_t cno, const OperatingPoint &op, double *upper_bound_perf, double *lower_bound_t) const¶
find an upper bound on the performance and a lower bound on t for configuration cno given another operating point op
-
void explore(Index *index, size_t nq, const float *xq, const AutoTuneCriterion &crit, OperatingPoints *ops) const¶
explore operating points
- Parameters:
index – index to run on
xq – query vectors (size nq * index.d)
crit – selection criterion
ops – resulting operating points
Public Members
-
std::vector<ParameterRange> parameter_ranges¶
all tunable parameters
-
int verbose¶
verbosity during exploration
-
int n_experiments¶
nb of experiments during optimization (0 = try all combinations)
-
size_t batchsize¶
maximum number of queries to submit at a time.
-
bool thread_over_batches¶
use multithreading over batches (useful to benchmark independent single-searches)
-
double min_test_duration¶
run tests several times until they reach at least this duration (to avoid jittering in MT mode)
-
virtual void initialize(const faiss::Index *index) override
-
struct GpuProgressiveDimIndexFactory : public faiss::ProgressiveDimIndexFactory
- #include <GpuCloner.h>
index factory for the ProgressiveDimClustering object
Public Functions
-
explicit GpuProgressiveDimIndexFactory(int ngpu)
-
virtual Index *operator()(int dim) override
ownership transferred to caller
-
virtual ~GpuProgressiveDimIndexFactory() override
Public Members
-
GpuMultipleClonerOptions options
-
std::vector<GpuResourcesProvider*> vres
-
std::vector<int> devices
-
int ncall
-
explicit GpuProgressiveDimIndexFactory(int ngpu)
-
class GpuResources
- #include <GpuResources.h>
Base class of GPU-side resource provider; hides provision of cuBLAS handles, CUDA streams and all device memory allocation performed
Subclassed by faiss::gpu::StandardGpuResourcesImpl
Public Functions
-
virtual ~GpuResources()
-
virtual void initializeForDevice(int device) = 0
Call to pre-allocate resources for a particular device. If this is not called, then resources will be allocated at the first time of demand
-
virtual cublasHandle_t getBlasHandle(int device) = 0
Returns the cuBLAS handle that we use for the given device.
-
virtual cudaStream_t getDefaultStream(int device) = 0
Returns the stream that we order all computation on for the given device
-
virtual void setDefaultStream(int device, cudaStream_t stream) = 0
Overrides the default stream for a device to the user-supplied stream. The resources object does not own this stream (i.e., it will not destroy it).
-
virtual std::vector<cudaStream_t> getAlternateStreams(int device) = 0
Returns the set of alternative streams that we use for the given device.
-
virtual void *allocMemory(const AllocRequest &req) = 0
Memory management Returns an allocation from the given memory space, ordered with respect to the given stream (i.e., the first user will be a kernel in this stream). All allocations are sized internally to be the next highest multiple of 16 bytes, and all allocations returned are guaranteed to be 16 byte aligned.
-
virtual void deallocMemory(int device, void *in) = 0
Returns a previous allocation.
-
virtual size_t getTempMemoryAvailable(int device) const = 0
For MemorySpace::Temporary, how much space is immediately available without cudaMalloc allocation?
-
virtual std::pair<void*, size_t> getPinnedMemory() = 0
Returns the available CPU pinned memory buffer.
-
virtual cudaStream_t getAsyncCopyStream(int device) = 0
Returns the stream on which we perform async CPU <-> GPU copies.
-
cublasHandle_t getBlasHandleCurrentDevice()
Calls getBlasHandle with the current device.
Functions provided by default
-
cudaStream_t getDefaultStreamCurrentDevice()
Calls getDefaultStream with the current device.
-
size_t getTempMemoryAvailableCurrentDevice() const
Calls getTempMemoryAvailable with the current device.
-
GpuMemoryReservation allocMemoryHandle(const AllocRequest &req)
Returns a temporary memory allocation via a RAII object.
-
void syncDefaultStream(int device)
Synchronizes the CPU with respect to the default stream for the given device
-
void syncDefaultStreamCurrentDevice()
Calls syncDefaultStream for the current device.
-
std::vector<cudaStream_t> getAlternateStreamsCurrentDevice()
Calls getAlternateStreams for the current device.
-
cudaStream_t getAsyncCopyStreamCurrentDevice()
Calls getAsyncCopyStream for the current device.
-
virtual ~GpuResources()
-
class GpuResourcesProvider
- #include <GpuResources.h>
Interface for a provider of a shared resources object.
Subclassed by faiss::gpu::StandardGpuResources
Public Functions
-
virtual ~GpuResourcesProvider()
-
virtual std::shared_ptr<GpuResources> getResources() = 0
Returns the shared resources object.
-
virtual ~GpuResourcesProvider()
-
template<typename GpuIndex>
struct IndexWrapper Public Functions
-
IndexWrapper(int numGpus, std::function<std::unique_ptr<GpuIndex>(GpuResourcesProvider*, int)> init)
-
void runOnIndices(std::function<void(GpuIndex*)> f)
-
void setNumProbes(int nprobe)
Public Members
-
std::vector<std::unique_ptr<faiss::gpu::StandardGpuResources>> resources
-
std::vector<std::unique_ptr<GpuIndex>> subIndex
-
std::unique_ptr<faiss::IndexReplicas> replicaIndex
-
IndexWrapper(int numGpus, std::function<std::unique_ptr<GpuIndex>(GpuResourcesProvider*, int)> init)
-
class KernelTimer
- #include <Timer.h>
Utility class for timing execution of a kernel.
Public Functions
-
KernelTimer(cudaStream_t stream = 0)
Constructor starts the timer and adds an event into the current device stream
-
~KernelTimer()
Destructor releases event resources.
-
float elapsedMilliseconds()
Adds a stop event then synchronizes on the stop event to get the actual GPU-side kernel timings for any kernels launched in the current stream. Returns the number of milliseconds elapsed. Can only be called once.
Private Members
-
cudaEvent_t startEvent_
-
cudaEvent_t stopEvent_
-
cudaStream_t stream_
-
bool valid_
-
KernelTimer(cudaStream_t stream = 0)
-
class StackDeviceMemory
- #include <StackDeviceMemory.h>
Device memory manager that provides temporary memory allocations out of a region of memory, for a single device
Public Functions
-
StackDeviceMemory(GpuResources *res, int device, size_t allocPerDevice)
Allocate a new region of memory that we manage.
-
StackDeviceMemory(int device, void *p, size_t size, bool isOwner)
Manage a region of memory for a particular device, with or without ownership
-
~StackDeviceMemory()
-
int getDevice() const
-
void *allocMemory(cudaStream_t stream, size_t size)
All allocations requested should be a multiple of 16 bytes.
-
void deallocMemory(int device, cudaStream_t, size_t size, void *p)
-
size_t getSizeAvailable() const
-
std::string toString() const
-
struct Range
- #include <StackDeviceMemory.h>
Previous allocation ranges and the streams for which synchronization is required
Public Functions
-
inline Range(char *s, char *e, cudaStream_t str)
Public Members
-
char *start_
-
char *end_
-
cudaStream_t stream_
-
inline Range(char *s, char *e, cudaStream_t str)
-
struct Stack
Public Functions
-
Stack(GpuResources *res, int device, size_t size)
Constructor that allocates memory via cudaMalloc.
-
~Stack()
-
size_t getSizeAvailable() const
Returns how much size is available for an allocation without calling cudaMalloc
-
char *getAlloc(size_t size, cudaStream_t stream)
Obtains an allocation; all allocations are guaranteed to be 16 byte aligned
-
void returnAlloc(char *p, size_t size, cudaStream_t stream)
Returns an allocation.
-
std::string toString() const
Returns the stack state.
Public Members
-
GpuResources *res_
Our GpuResources object.
-
int device_
Device this allocation is on.
-
char *alloc_
Where our temporary memory buffer is allocated; we allocate starting 16 bytes into this
-
size_t allocSize_
Total size of our allocation.
-
char *start_
Our temporary memory region; [start_, end_) is valid.
-
char *end_
-
char *head_
Stack head within [start, end)
-
std::list<Range> lastUsers_
List of previous last users of allocations on our stack, for possible synchronization purposes
-
size_t highWaterMemoryUsed_
What’s the high water mark in terms of memory used from the temporary buffer?
-
Stack(GpuResources *res, int device, size_t size)
-
StackDeviceMemory(GpuResources *res, int device, size_t allocPerDevice)
-
class StandardGpuResources : public faiss::gpu::GpuResourcesProvider
- #include <StandardGpuResources.h>
Default implementation of GpuResources that allocates a cuBLAS stream and 2 streams for use, as well as temporary memory. Internally, the Faiss GPU code uses the instance managed by getResources, but this is the user-facing object that is internally reference counted.
Public Functions
-
StandardGpuResources()
-
~StandardGpuResources() override
-
virtual std::shared_ptr<GpuResources> getResources() override
Returns the shared resources object.
-
void noTempMemory()
Disable allocation of temporary memory; all temporary memory requests will call cudaMalloc / cudaFree at the point of use
-
void setTempMemory(size_t size)
Specify that we wish to use a certain fixed size of memory on all devices as temporary memory. This is the upper bound for the GPU memory that we will reserve. We will never go above 1.5 GiB on any GPU; smaller GPUs (with <= 4 GiB or <= 8 GiB) will use less memory than that. To avoid any temporary memory allocation, pass 0.
-
void setPinnedMemory(size_t size)
Set amount of pinned memory to allocate, for async GPU <-> CPU transfers
-
void setDefaultStream(int device, cudaStream_t stream)
Called to change the stream for work ordering. We do not own
stream; i.e., it will not be destroyed when the GpuResources object gets cleaned up. We are guaranteed that all Faiss GPU work is ordered with respect to this stream upon exit from an index or other Faiss GPU call.
-
void revertDefaultStream(int device)
Revert the default stream to the original stream managed by this resources object, in case someone called
setDefaultStream.
-
void setDefaultNullStreamAllDevices()
Called to change the work ordering streams to the null stream for all devices
-
std::map<int, std::map<std::string, std::pair<int, size_t>>> getMemoryInfo() const
Export a description of memory used for Python.
-
cudaStream_t getDefaultStream(int device)
Returns the current default stream.
-
size_t getTempMemoryAvailable(int device) const
Returns the current amount of temp memory available.
-
void syncDefaultStreamCurrentDevice()
Synchronize our default stream with the CPU.
-
void setLogMemoryAllocations(bool enable)
If enabled, will print every GPU memory allocation and deallocation to standard output
Private Members
-
std::shared_ptr<StandardGpuResourcesImpl> res_
-
StandardGpuResources()
-
class StandardGpuResourcesImpl : public faiss::gpu::GpuResources
- #include <StandardGpuResources.h>
Standard implementation of the GpuResources object that provides for a temporary memory manager
Public Functions
-
StandardGpuResourcesImpl()
-
~StandardGpuResourcesImpl() override
-
void noTempMemory()
Disable allocation of temporary memory; all temporary memory requests will call cudaMalloc / cudaFree at the point of use
-
void setTempMemory(size_t size)
Specify that we wish to use a certain fixed size of memory on all devices as temporary memory. This is the upper bound for the GPU memory that we will reserve. We will never go above 1.5 GiB on any GPU; smaller GPUs (with <= 4 GiB or <= 8 GiB) will use less memory than that. To avoid any temporary memory allocation, pass 0.
-
void setPinnedMemory(size_t size)
Set amount of pinned memory to allocate, for async GPU <-> CPU transfers
-
virtual void setDefaultStream(int device, cudaStream_t stream) override
Called to change the stream for work ordering. We do not own
stream; i.e., it will not be destroyed when the GpuResources object gets cleaned up. We are guaranteed that all Faiss GPU work is ordered with respect to this stream upon exit from an index or other Faiss GPU call.
-
void revertDefaultStream(int device)
Revert the default stream to the original stream managed by this resources object, in case someone called
setDefaultStream.
-
virtual cudaStream_t getDefaultStream(int device) override
Returns the stream for the given device on which all Faiss GPU work is ordered. We are guaranteed that all Faiss GPU work is ordered with respect to this stream upon exit from an index or other Faiss GPU call.
-
void setDefaultNullStreamAllDevices()
Called to change the work ordering streams to the null stream for all devices
-
void setLogMemoryAllocations(bool enable)
If enabled, will print every GPU memory allocation and deallocation to standard output
-
virtual void initializeForDevice(int device) override
Internal system calls.
Initialize resources for this device
-
virtual cublasHandle_t getBlasHandle(int device) override
Returns the cuBLAS handle that we use for the given device.
-
virtual std::vector<cudaStream_t> getAlternateStreams(int device) override
Returns the set of alternative streams that we use for the given device.
-
virtual void *allocMemory(const AllocRequest &req) override
Allocate non-temporary GPU memory.
-
virtual void deallocMemory(int device, void *in) override
Returns a previous allocation.
-
virtual size_t getTempMemoryAvailable(int device) const override
For MemorySpace::Temporary, how much space is immediately available without cudaMalloc allocation?
-
std::map<int, std::map<std::string, std::pair<int, size_t>>> getMemoryInfo() const
Export a description of memory used for Python.
-
virtual std::pair<void*, size_t> getPinnedMemory() override
Returns the available CPU pinned memory buffer.
-
virtual cudaStream_t getAsyncCopyStream(int device) override
Returns the stream on which we perform async CPU <-> GPU copies.
-
cublasHandle_t getBlasHandleCurrentDevice()
Calls getBlasHandle with the current device.
Functions provided by default
-
cudaStream_t getDefaultStreamCurrentDevice()
Calls getDefaultStream with the current device.
-
size_t getTempMemoryAvailableCurrentDevice() const
Calls getTempMemoryAvailable with the current device.
-
GpuMemoryReservation allocMemoryHandle(const AllocRequest &req)
Returns a temporary memory allocation via a RAII object.
-
void syncDefaultStream(int device)
Synchronizes the CPU with respect to the default stream for the given device
-
void syncDefaultStreamCurrentDevice()
Calls syncDefaultStream for the current device.
-
std::vector<cudaStream_t> getAlternateStreamsCurrentDevice()
Calls getAlternateStreams for the current device.
-
cudaStream_t getAsyncCopyStreamCurrentDevice()
Calls getAsyncCopyStream for the current device.
Private Functions
-
bool isInitialized(int device) const
Have GPU resources been initialized for this device yet?
Private Members
-
std::unordered_map<int, std::unordered_map<void*, AllocRequest>> allocs_
Set of currently outstanding memory allocations per device device -> (alloc request, allocated ptr)
-
std::unordered_map<int, std::unique_ptr<StackDeviceMemory>> tempMemory_
Temporary memory provider, per each device.
-
std::unordered_map<int, cudaStream_t> defaultStreams_
Our default stream that work is ordered on, one per each device.
-
std::unordered_map<int, cudaStream_t> userDefaultStreams_
This contains particular streams as set by the user for ordering, if any
-
std::unordered_map<int, std::vector<cudaStream_t>> alternateStreams_
Other streams we can use, per each device.
-
std::unordered_map<int, cudaStream_t> asyncCopyStreams_
Async copy stream to use for GPU <-> CPU pinned memory copies.
-
std::unordered_map<int, cublasHandle_t> blasHandles_
cuBLAS handle for each device
-
void *pinnedMemAlloc_
Pinned memory allocation for use with this GPU.
-
size_t pinnedMemAllocSize_
-
size_t tempMemSize_
Another option is to use a specified amount of memory on all devices
-
size_t pinnedMemSize_
Amount of pinned memory we should allocate.
-
bool allocLogging_
Whether or not we log every GPU memory allocation and deallocation.
Private Static Functions
-
static size_t getDefaultTempMemForGPU(int device, size_t requested)
Adjust the default temporary memory allocation based on the total GPU memory size
-
StandardGpuResourcesImpl()
-
struct ToCPUCloner : public faiss::Cloner
- #include <GpuCloner.h>
Cloner specialized for GPU -> CPU.
Public Functions
-
virtual VectorTransform *clone_VectorTransform(const VectorTransform*)¶
-
virtual VectorTransform *clone_VectorTransform(const VectorTransform*)¶
-
struct ToGpuCloner : public faiss::Cloner, public faiss::gpu::GpuClonerOptions
- #include <GpuCloner.h>
Cloner specialized for CPU -> 1 GPU.
Public Functions
-
ToGpuCloner(GpuResourcesProvider *prov, int device, const GpuClonerOptions &options)
-
virtual VectorTransform *clone_VectorTransform(const VectorTransform*)¶
Public Members
-
GpuResourcesProvider *provider
-
int device
-
IndicesOptions indicesOptions¶
how should indices be stored on index types that support indices (anything but GpuIndexFlat*)?
-
bool useFloat16CoarseQuantizer¶
is the coarse quantizer in float16?
-
bool useFloat16¶
for GpuIndexIVFFlat, is storage in float16? for GpuIndexIVFPQ, are intermediate calculations in float16?
-
bool usePrecomputed¶
use precomputed tables?
-
long reserveVecs¶
reserve vectors in the invfiles?
-
bool storeTransposed¶
For GpuIndexFlat, store data in transposed layout?
-
bool verbose¶
Set verbose options on the index.
-
ToGpuCloner(GpuResourcesProvider *prov, int device, const GpuClonerOptions &options)
-
struct ToGpuClonerMultiple : public faiss::Cloner, public faiss::gpu::GpuMultipleClonerOptions
- #include <GpuCloner.h>
Cloner specialized for CPU -> multiple GPUs.
Public Functions
-
ToGpuClonerMultiple(std::vector<GpuResourcesProvider*> &provider, std::vector<int> &devices, const GpuMultipleClonerOptions &options)
-
ToGpuClonerMultiple(const std::vector<ToGpuCloner> &sub_cloners, const GpuMultipleClonerOptions &options)
-
virtual VectorTransform *clone_VectorTransform(const VectorTransform*)¶
Public Members
-
std::vector<ToGpuCloner> sub_cloners
-
bool shard¶
Whether to shard the index across GPUs, versus replication across GPUs
-
int shard_type¶
IndexIVF::copy_subset_to subset type.
-
IndicesOptions indicesOptions¶
how should indices be stored on index types that support indices (anything but GpuIndexFlat*)?
-
bool useFloat16CoarseQuantizer¶
is the coarse quantizer in float16?
-
bool useFloat16¶
for GpuIndexIVFFlat, is storage in float16? for GpuIndexIVFPQ, are intermediate calculations in float16?
-
bool usePrecomputed¶
use precomputed tables?
-
long reserveVecs¶
reserve vectors in the invfiles?
-
bool storeTransposed¶
For GpuIndexFlat, store data in transposed layout?
-
bool verbose¶
Set verbose options on the index.
-
ToGpuClonerMultiple(std::vector<GpuResourcesProvider*> &provider, std::vector<int> &devices, const GpuMultipleClonerOptions &options)
-
namespace utils
Functions
- template<typename U, typename V> constexpr __host__ __device__ auto divDown (U a, V b) -> decltype(a+b)
- template<typename U, typename V> constexpr __host__ __device__ auto divUp (U a, V b) -> decltype(a+b)
- template<typename U, typename V> constexpr __host__ __device__ auto roundDown (U a, V b) -> decltype(a+b)
- template<typename U, typename V> constexpr __host__ __device__ auto roundUp (U a, V b) -> decltype(a+b)
- template<class T> constexpr __host__ __device__ T pow (T n, T power)
- template<class T> constexpr __host__ __device__ T pow2 (T n)
- template<typename T> constexpr __host__ __device__ int log2 (T n, int p=0)
- template<typename T> constexpr __host__ __device__ bool isPowerOf2 (T v)
- template<typename T> constexpr __host__ __device__ T nextHighestPowerOf2 (T v)
-
enum class DistanceDataType
-
namespace ivflib
-
Functions
-
void check_compatible_for_merge(const Index *index1, const Index *index2)
check if two indexes have the same parameters and are trained in the same way, otherwise throw.
-
const IndexIVF *extract_index_ivf(const Index *index)
get an IndexIVF from an index. The index may be an IndexIVF or some wrapper class that encloses an IndexIVF
throws an exception if this is not the case.
-
const IndexIVF *try_extract_index_ivf(const Index *index)
same as above but returns nullptr instead of throwing on failure
-
void merge_into(Index *index0, Index *index1, bool shift_ids)
Merge index1 into index0. Works on IndexIVF’s and IndexIVF’s embedded in a IndexPreTransform. On output, the index1 is empty.
- Parameters:
shift_ids – translate the ids from index1 to index0->prev_ntotal
-
void search_and_return_centroids(Index *index, size_t n, const float *xin, long k, float *distances, idx_t *labels, idx_t *query_centroid_ids, idx_t *result_centroid_ids)
-
ArrayInvertedLists *get_invlist_range(const Index *index, long i0, long i1)
Get a subset of inverted lists [i0, i1)
-
void set_invlist_range(Index *index, long i0, long i1, ArrayInvertedLists *src)
Set a subset of inverted lists.
-
void search_with_parameters(const Index *index, idx_t n, const float *x, idx_t k, float *distances, idx_t *labels, const IVFSearchParameters *params, size_t *nb_dis = nullptr, double *ms_per_stage = nullptr)
search an IndexIVF, possibly embedded in an IndexPreTransform with given parameters. This is a way to set the nprobe and get statdistics in a thread-safe way.
Optionally returns (if non-nullptr):
nb_dis: number of distances computed
ms_per_stage: [0]: preprocessing time [1]: coarse quantization, [2]: list scanning
-
void range_search_with_parameters(const Index *index, idx_t n, const float *x, float radius, RangeSearchResult *result, const IVFSearchParameters *params, size_t *nb_dis = nullptr, double *ms_per_stage = nullptr)
same as search_with_parameters but for range search
-
IndexIVFResidualQuantizer *ivf_residual_from_quantizer(const ResidualQuantizer&, int nlevel)
Build an IndexIVFResidualQuantizer from an ResidualQuantizer, using the nlevel first components as coarse quantizer and the rest as codes in invlists
-
void ivf_residual_add_from_flat_codes(IndexIVFResidualQuantizer *ivfrq, size_t ncode, const uint8_t *codes, int64_t code_size = -1)
add from codes. NB that the norm component is not used, so the code_size can be provided.
- Parameters:
ivfrq – index to populate with the codes
codes – codes to add, size (ncode, code_size)
code_size – override the ivfrq’s code_size, useful if the norm encoding is different
-
struct SlidingIndexWindow
- #include <IVFlib.h>
A set of IndexIVFs concatenated together in a FIFO fashion. at each “step”, the oldest index slice is removed and a new index is added.
Public Functions
-
SlidingIndexWindow(Index *index)
index should be initially empty and trained
-
void step(const Index *sub_index, bool remove_oldest)
Add one index to the current index and remove the oldest one.
- Parameters:
sub_index – slice to swap in (can be NULL)
remove_oldest – if true, remove the oldest slices
Public Members
-
Index *index
common index that contains the sliding window
-
ArrayInvertedLists *ils
InvertedLists of index.
-
int n_slice
number of slices currently in index
-
size_t nlist
same as index->nlist
-
std::vector<std::vector<size_t>> sizes
cumulative list sizes at each slice
-
SlidingIndexWindow(Index *index)
-
void check_compatible_for_merge(const Index *index1, const Index *index2)
-
namespace quantize_lut
Functions to quantize PQ floating-point Look Up Tables (LUT) to uint8, and biases to uint16. The accumulation is supposed to take place in uint16. The quantization coefficients are float (a, b) such that
original_value = quantized_value * a / b
The hardest part of the quantization is with multiple LUTs that need to be added up together. In that case, coefficient a has to be chosen so that the sum fits in a uint16 accumulator.
Functions
-
void round_uint8_per_column(float *tab, size_t n, size_t d, float *a_out = nullptr, float *b_out = nullptr)
-
void round_uint8_per_column_multi(float *tab, size_t m, size_t n, size_t d, float *a_out = nullptr, float *b_out = nullptr)
-
void quantize_LUT_and_bias(size_t nprobe, size_t M, size_t ksub, bool lut_is_3d, const float *LUT, const float *bias, uint8_t *LUTq, size_t M2, uint16_t *biasq, float *a_out = nullptr, float *b_out = nullptr)
LUT quantization to uint8 and bias to uint16.
(nprobe, M, ksub, lut_is_3d) determine the size of the the LUT
LUT input:
2D size (M, ksub): single matrix per probe (lut_is_3d=false)
3D size (nprobe, M, ksub): separate LUT per probe (lut_is_3d=true) bias input:
nullptr: bias is 0
size (nprobe): one bias per probe Output:
LUTq uint8 version of the LUT (M size is rounded up to M2)
biasq (or nullptr): uint16 version of the LUT
a, b: scalars to approximate the true distance
-
void aq_quantize_LUT_and_bias(size_t nprobe, size_t M, size_t ksub, const float *LUT, const float *bias, size_t M_norm, int norm_scale, uint8_t *LUTq, size_t M2, uint16_t *biasq, float *a_out, float *b_out)
-
float aq_estimate_norm_scale(size_t M, size_t ksub, size_t M_norm, const float *LUT)
-
void round_uint8_per_column(float *tab, size_t n, size_t d, float *a_out = nullptr, float *b_out = nullptr)
-
using IndexReplicas = IndexReplicasTemplate<Index>